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Preface to the Classics Edition 


An accessible text for the study of numerical methods for solving least 
squares problems remains an essential part of a scientific software foundation. 
Feedback that we have received from practicing engineers and scientists, as well 
as from educators and students in numerical analysis, indicates that this book 
has served this purpose. We were pleased when SIAM decided to republish the 
book in their Classics in Applied Mathematics series. 

The main body of the book remains unchanged from the original book that 
was published by Prentice-Hall in 1974, with the exception of corrections to 
known errata. Appendix C has been edited to reflect changes in the associated 
software package and the software distribution method. A new Appendix D 
has been added, giving a brief survey of the many new developments in topics 
treated in the book during the period 1974-1995. Appendix D is organized 
into sections corresponding to the chapters of the main body of the book and 
includes a bibliography listing about 230 publications from 1974 to 1995. 

The 1974 book was accompanied by a set of Fortran 66 subroutines that 
implemented about ten of the major algorithms described in the book and six 
sample main programs that illustrated the use of the subroutines. The codes 
were listed in full in the book and distributed first by IMSL, Inc. (now Visual 
Numerics, Inc.) of Houston, Texas, and later by C Abaci, Inc. This software 
was tested originally on computers of the major vendors of the time and has 
been tested further on a wide range of personal computers, workstations, and 
supercomputers. 

The software from the original book has been upgraded to conform to the 
Fortran 77 standard to accompany this SIAM edition. A new subroutine, 
BVLS, has been added for solution of the Bounded Variables Least Squares 
problem, which is a problem that was not specifically discussed in the original 
book. Subroutine BVLS is written in Fortran 90. 

For this edition, codes are not listed in the book but rather are being made 
available from NETLIB via the Internet. We plan a future augmentation of 
the software package in NETLIB with versions in the ANSI C and Fortran 90 
languages. 

We again wish to thank Gene Golub and the late George Forsythe for sug¬ 
gesting to us in about 1969-1970 the writing of the original edition and for giving 
us encouragement throughout the process. Gene again gave significant impetus 
to the idea of the current SIAM republication. We are grateful to our colleagues 
around the world who examined a draft of Appendix D that we made available 
via the Internet and contributed improvements and bibliographic citations, also 
via the Internet. We thank Vickie Keam of SIAM for her cheerful patience in 
working out the details of publishing the new edition. Finally we express grati¬ 
tude to our friends and colleagues Fred Krogh and W. Van Snyder for the many 
personal favors they provided and for adding constructive comments about our 
writing and codes. 


xi 
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PREFACE 


This book brings together a body of information on solving least squares 
problems whose practical development has taken place mainly during the past 
decade. This information is valuable to the scientist, engineer, or student who 
must analyze and solve systems of linear algebraic equations. These systems 
may be overdetermined, underdetermined, or exactly determined and may or 
may not be consistent. They may also include both linear equality and ine¬ 
quality constraints. 

Practitioners in specific fields have developed techniques and nomen¬ 
clature for the least squares problems of their own discipline. The material 
presented in this book can unify this divergence of methods. 

Essentially all real problems are nonlinear. Many of the methods of 
reaching an understanding of nonlinear problems or computing using non¬ 
linear models involve the local replacement of the nonlinear problem by a 
linear one. Specifically, various methods of analyzing and solving the non¬ 
linear least squares problem involve solving a sequence of linear least squares 
problems. One essential requirement of these methods is the capability of 
computing solutions to (possibly poorly determined) linear least squares 
problems which are reasonable in the context of the nonlinear problem. 

For the reader whose immediate concern is with a particular application 
we suggest first reading Chapter 25. Following this the reader may find that 
some of the Fortran programs and subroutines of Appendix C have direct 
applicability to his problem. 

There have been many contributors to theoretical and practical develop¬ 
ment of least squares techniques since the early work of Gauss. We particu¬ 
larly cite Professor G. H. Golub who, through broad involvement in both 
theoretical and practical aspects of the problem, has contributed many signi¬ 
ficant ideas and algorithms relating to the least squares problem. 

The first author introduced the second author to the least squares problem 

xiii 
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in 1966. Since that time the authors have worked very closely together in the 
adaptation of the stable mathematical methods to practical computational 
problems at JPL. This book was written as part of this collaborative effort. 

We wish to express our appreciation to the late Professor G. E. Forsythe 
for extending to us the opportunity of writing this book. We thank the mana¬ 
gement of the Jet Propulsion Laboratory, California Institute of Technology, 
for their encouragement and support. We thank Drs. F. T. Krogh and D. 
Carta for reading the manuscript and providing constructive comments. A 
number of improvements and simplifications of mathematical proofs were 
due to Dr. Krogh. 

Our thanks go to Mrs. Roberta Cerami for her careful and cheerful 
typing of the manuscript in all of its stages. 

Charles L. Lawson 
Richard J. Hanson 



INTRODUCTION 


This book is intended to be used as both a text and a reference for persons 
who are investigating the solutions of linear least squares problems. Such 
least squares problems often occur as a component part of some larger com¬ 
putational problem. As an example, the determination of the orbit of a space¬ 
craft is often considered mathematically as a multipoint boundary value 
problem in the field of ordinary differential equations, but the computation 
of the orbital parameters usually amounts to a nonlinear least squares esti¬ 
mation problem using various linearization schemes. 

More generally, almost any problem with sufficient data to overdetermine 
a solution calls for some type of approximation method. Most frequently 
least squares is the approximation criterion chosen. 

There are a number of auxiliary requirements which often arise in least 
squares computations which merit identification for algorithmic develop¬ 
ment. Examples: 

A problem may require certain equality or inequality relations between 
variables. A problem may involve very large volumes of data so that the 
allocation of computer storage is of major concern. 

Many times the purpose of least squares computation is not merely to 
find some set of numbers that “solve” the problem, but rather the investigator 
wishes to obtain additional quantitative information describing the relation¬ 
ship of the solution parameters to the data. In particular, there may be a 
family of different solution parameters that satisfy the stated conditions 
almost equally well and the investigator may wish to identify this indeter¬ 
minacy and make a choice in this family to meet some additional condi¬ 
tions. 

This book presents numerical methods for obtaining least squares solu- 
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tions keeping the points above in mind. These methods have been used suc¬ 
cessfully by many engineers and scientists, together with the authors, in the 
NASA unmanned space program. 

The least squares problem that we are considering here is known by 
different names by those in different scientific disciplines. For example, 
mathematicians may regard the (least squares) problem as finding the closest 
point in a given subspace to a given point in a function space. Numerical 
analysts have also tended to use this framework, which tends to ignore data 
errors. It follows that the opportunity to exploit the often present arbitrari¬ 
ness of the solution is disregarded. 

Statisticians introduce probability distributions into their conception 
of the problem and use such terms as regression analysis to describe this 
area. Engineers reach this problem by studying such topics as parameter 
estimation, filtering, or process identification. 

The salient point is this: When these problems (formulated in any of 
these contexts) reach the stage of numerical computation, they contain the 
same central problem, namely, a sequence of linear least squares systems. 

This basic linear least squares problem can be stated as follows: 

Problem LS 

Given a real m x n matrix A of rank k < min (m, n), and given a real 
m -vector b,find a real n -vector x 0 minimizing the euclidean length of Ax — b. 

(The reader can refer to Appendix A for the definitions of any unfamiliar 
linear algebraic terms.) We shall use the symbolism Ax^b to denote 
Problem LS. 

This problem can also be posed and studied for the case of complex A 
and b. The complex case arises much less frequently in practice than the real 
case and the theory and computational methods for the real case generalize 
quite directly to the complex case. 

In addition to the statement of Problem LS (in a given context) there 
is an additional condition: The numerical data that constitute A and b have 
only a limited number of accurate digits and after those digits the data are 
completely uncertain and hence arbitrary ! In practice this is the usual state 
of affairs. This is due, in part, to the limited accuracy of measurements or 
observations. It is important that explicit note be taken of this situation and 
that it be used to advantage to obtain an appropriate approximate solution 
to the problem. We shall discuss methods for accomplishing this, particularly 
in Chapters 25 and 26. 

Consider, as an important example, the case in which the linear least 
squares problem arises from a nonlinear one and the solution vector x 0 is 
to be used as a correction to be added to a current nominal solution of the 
nonlinear problem. The linearized problem will be a useful approximation 



CASE 1o 


CASE 1b 




Ax a b Ax 3 b 

HANK (A) - m-ti RANK (A) « k < m*n 


CASE 2o 



CASE 2b 




Ax * b Ax % b 

RANK (A) * ivcm RANK (A) = k < nx m 


CASE 3a 




CASE 3b 


m<n 


Ax * b Ax * b 

RANK (A) * m < n RANK (A) * k < m < n 

Fig. 1.1 The six cases of Problem LS depending on the relative 
sizes of m, n, and Rank (A). 
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to the nonlinear one only in some limited neighborhood. If there are differ¬ 
ent vectors that give satisfactorily small residuals in the linear problem, one 
may prefer the one of smallest length to increase the likelihood of staying 
within the neighborhood of validity of the linear problem. 

The following three points are fundamental to our approach to least 
squares computation. 

1. Since the data defining Problem LS are uncertain, we can change 
these data to suit our needs within this uncertainty. 

2. We shall use orthogonal elimination matrices directly on the linear 
system of Problem LS. 

3. Practicality, for computer implementation, is stressed throughout. 

We shall briefly comment on the first two of these points. Our goal in 
changing the problem, as stated in point 1, will be to avoid the situation above 
where a “small” change in the data produces “large” changes in the solution. 

Orthogonal transformation matrices, mentioned in point 2, have a natural 
place in least squares computations because they leave the euclidean length 
of a vector invariant. Furthermore, it is desirable to use them because of their 
stability in computation with regard to propagating data errors or uncer¬ 
tainty. 

We have made no assumption regarding the relative sizes of m and it. 
In our subsequent discussion of Problem LS it will be convenient to identify 
the six cases illustrated in Fig. 1.1. 

The main interest of this book will be in Case 2a with special attention 
to the situation where data uncertainty leads to Case 2b. Algorithms and 
discussions applicable to all six cases will be given, however. 



2 ANALYSIS OF THE 

LEAST SQUARES PROBLEM 


The central point of this chapter will be an analysis of Problem LS based 
on a certain decomposition of an m x n matrix A in the form HRK T where 
H and K are orthogonal matrices. The definition of an orthogonal matrix 
and other linear algebraic concepts are summarized in Appendix A. 

Our interest in this general type of decomposition A — HRK T is moti¬ 
vated by the practicality and usefulness of certain specific computable decom¬ 
positions of this type that are introduced in Chapters 3 and 4. 

An important property of orthogonal matrices is the preservation of 
euclidean length under multiplication. Thus for any m-vector y and any 
m x m orthogonal matrix Q, 

(2.1) II Orll-ll?ll 

In the context of Problem LS, the least squares problem of minimizing 
the euclidean length of Ax — b, we have 

(2.2) |! Q{Ax - 6)|| -1| QAx - Qb\\ = || Ax - *|| 

for any m x m orthogonal matrix Q and any n-vector x. 

The use of orthogonal transformations allows us to express the solution 
of Problem LS in the following way: 

(2.3) Theorem 

Suppose that A is anm x a matrix of rank k and that 

(2.4) A = HRK T 
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where 

(a) H is an m x m orthogonal matrix. 

(b) R is an m x n matrix of the form 



(c) R m is a k x k matrix of rank k. 

(d) K is an n x n orthogonal matrix. 

Define the vector 


(2.5) 


Lft 


\}m-k 


and introduce the new variable 


( 2 . 6 ) 


L^iJl» - 


Define y, to be the unique solution of 



(1) Then all solutions to the problem of minimizing |[ Ax — b|| are of 
the form 





where y 2 is arbitrary. 

(2) Any such i gives rise to the same residual vector r satisfying 


(2.9) 


r = b — Ag — 



(3) The norm of r satisfies 


(2.10) ||r|| = ||*-itt||«||f 1 || 

(4) The unique solution of minimum length is 
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Proof: Replacing A by the right side of Eq. (2.4) and using Eq. (2.2) 

yields 


(2.12) \]Ax — b H* = || HRKfx - b ||* 

= WRfFx - /Fb||* 

Using Eq. (2.S) to (2.8) we see that 

(2.13) \\Ax-b\\* = \\R uyi -g,||* + ||fall* 
for all x. 

The right member of Eq. (2.13) has the minimum value Hffxll 1 when 

(2.14) R\iyi — g\ 

Equation (2.14) has a unique solution y, since J?,, is of rank k. 

The most general solution for y is given by 


(2.15) 




where y 2 is arbitrary. 

Then with X defined by Eq. (2.8) one obtains 


b — AX = b — HRKTX = H(g — Ry) = H ° 

L?*J 


which establishes Eq. (2.9). 

Clearly the minimum length vector y, of the form given in Eq. (2.15), 
is that one with y 2 — 0. From Eq. (2.8) it follows that the solution x of mini¬ 
mum euclidean length is given by 


(2.16) 



This completes the proof of Theorem (2.3). 


Any decomposition of an m x n matrix A = HRK T , as in Eq. (2.4), 
will be called an orthogonal decomposition of A. 

In case k = n or k — m, quantities having dimensions involving (n — k) 
or ( m — k) are respectively absent. In particular, the solution to Problem 
LS is unique in case k = n. 

Notice that the solution of minimum length, the set of all solutions, 
and the minimum value, for the problem of minimizing \\Ax — b\\, ate alt 
unique; they do not depend on the particular orthogonal decomposition. 
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EXERCISE 

(2.17) Suppose A mxm m ^ n, is of rank n with Q x R t — A — Q X R X , If the 
Qt are m x n with orthogonal column vectors and the it, are n x n 
upper triangular, then there exists a diagonal matrix D with diagonal 
elements plus or minus one such that Q t D — Q t and DR t — R t . 



ORTHOGONAL DECOMPOSITION 

3 BY CERTAIN ELEMENTARY 

ORTHOGONAL TRANSFORMATIONS 


In the previous chapter it was shown that Problem LS could be solved 
assuming an orthogonal decomposition of the m x n matrix A. In this chap¬ 
ter we shall establish the existence of such a decomposition for any matrix 
A using explicit orthogonal transformations. 

Two elementary orthogonal transformations that will be useful are pre¬ 
sented in the following two lemmas. 

(3.1) L£MMA 

Given an m- vector v (which is not the zero vector), there exists an orthog¬ 
onal matrix Q such that 


(3.2) 

with 


and 


(?» = —er ||w||e. 



+1 i / v ,^0 

-1 i/v, <0 


where v t is the first component of v. 
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Proof: Define 


and 

(3.3) 


m = » + er ii v || e, 


Q = L 


2uu T 

U T U 


The proof is completed by verifying directly that the matrix Q of Eq. 
(3.3) is symmetric and orthogonal and that the remaining statement of Lem¬ 
ma (3.1) is satisfied. 


The transformation defined in Eq. (3.3) was used by A. S. Householder 
in discussing certain eigenvalue problems [Householder (1958)]. For this 
reason the matrix Q is called a Householder transformation matrix. 

This transformation can be viewed geometrically as a reflection in the 
(m — l)-dimensional subspace, 5, orthogonal to the vector u. By this it is 
meant that Qu = — u and Qs~s for all s e S. 

In the special case that one wishes to transform only one element of the 
vector v to zero, the following Givens transformation [Givens (1954)] is often 
used. Since this transformation only modifies two components of a vector, 
it suffices to discuss its operation on a 2-vector. 

(3.4) Lemma 

Given a 2-vector v = (v„ v,) T (with either v, & 0 or \ 2 & 0), there 
is a 2 X 2 orthogonal matrix. 

<3! » ®-U 3 

with 



such that 


c* + s* = 1 


(3.7) 


Gv 




0 


Proof: Simply put 


(3.8) 
and 

(3.9) 


c “ (*» +v» 



Vi 

(v\ + »\Y n 



CHAP. 3 


CERTAIN ELEMENTARY ORTHOGONAL TRANSFORMATIONS 11 


Then the matrix G of Eq. (3.5) can be easily shown to be orthogonal 
and satisfy Eq. (3.7), completing the proof of Lemma (3.4). 

We also remark that we may choose G to be symmetric as well as orthogo¬ 
nal by defining 

(MO, G =(; _') 

with c and s as in Eq. (3.8) and (3.9). 

(3.11) Theorem 

Let A be an m x n matrix. There is an m X m orthogonal matrix 
Q such that QA = R is zero below the main diagonal. 

Let an m x m orthogonal matrix Q, be chosen as in Lemma (3.1) so 
that the first column of Q,A is zeroed in components 2 through m. Next 
choose an (m — 1) x (m — 1) orthogonal matrix P ly which when applied 
to the (m — 1) vector consisting of components 2 through m of the second 
column of Q,A results in zeros in components 3 through m. The transforma¬ 
tion matrix 



is orthogonal and Q 2 Q X A has zeros below the diagonal in both the first two 
columns. 

Continuing in this way, a product of at most n orthogonal transforma¬ 
tions can be constructed that will transform A to upper triangular form. 
These remarks can be formalized to provide a finite induction proof of Theo¬ 
rem (3.11). 

The algorithmic details of the construction of these transformations will 
be given in Chapter 10. This decomposition of a matrix as a product of an 
orthogonal matrix and an upper triangular matrix is called a QR decomposi¬ 
tion. It plays an important role in a number of computational algorithms 
of linear algebra [Francis (1961); Golub and Businger (1965)). 

For Cases la and 2a ofFig. 1.1, in which Rank (A) — n, Theorem (3.11) 
establishes the existence of an orthogonal decomposition of A. Thus from 
Theorem (3.11) we can write 

(3.12) A = Q T R=~ Q T RI. 

where the matrices Q T , R, and /„ in this representation have the properties 
required of the matrices H, R, and K T of an orthogonal decomposition for 
a matrix A, as given in Theorem (2.3). 
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In the case that the rank of A is m, Case 3a of Fig. 1.1, Theorem (3.11) 
allows us to write 

(3.13) A T = (FR 

so that 


(3.14) 


A = RFQ = I m lFQ 


Here, I m , IF, and Q in this representation have the properties required of 
the matrices H, R, and K T for an orthogonal decomposition of a matrix of 
rank m, as given in Theorem (2.3). 

In Cases lb, 2b, and 3b of Fig. 1.1, the matrix R obtained in Theorem 
(3.11) is not necessarily in the form required for an orthogonal decomposi¬ 
tion. 

We proceed to discuss additional transformations that will obtain the 
orthogonal decomposition for these cases. 

(3.15) Theorem 

Let A be an m x n matrix whose rank k satisfies k < n m. There 
is an m x m orthogonal matrix Q and an n x n permutation matrix 
P such that 


(3.16) 



Rkxk 

Q(m-k)xk 


*(*-«_ 


Here R « a k x k upper triangular matrix of rank k. 


Proof: Let the permutation matrix P be selected so that the first 
k columns of the matrix AP are linearly independent. By application of 
Theorem (3.11) there is an m x m orthogonal matrix such that QAP is upper 
triangular. Since the first k columns of AP are linearly independent, the 
same will be true of the first k columns of QAP. 

All the elements in rows k + 1 through m of columns k + 1 through 
n of QAP will be zero. Otherwise the rank of QAP would be greater than k, 
which would contradict the fact that A is of rank k. Therefore QAP has 
the form indicated in the right member of Eq. (3.16). This completes the 
proof of Theorem (3.15). 


The submatrix [/?: T\ in the right member of Eq. (3.16) can be further 
transformed to the compact form required of the matrix R in Theorem (2.3). 
This transformation is described by the following lemma: 



CHAP. 3 


CERTAIN ELEMENTARY ORTHOGONAL TRANSFORMATIONS 13 


(3.17) Lemma 

Let [R: TJkakxn matrix where R is of rank k. There is an n x n 
orthogonal matrix W such that 


(3.18) [R: T)W = 0* xtl> _*,] 

Here R is a lower triangular matrix of rank k. 

Lemma (3.17) follows from Theorem (3.IS) by identification of the 
present n, k, [it: T\, and W , respectively, with m, n, A T , and Q T of Theorem 
(3.15). 

Lemma (3.17) can be used together with Theorem (3.15) to prove the 
following theorem: 

(3.19) Theorem 

Let A be an m x n matrix of rank k. Then there is an m x m ortho¬ 
gonal matrix H and an n x n orthogonal matrix K such that 

(3.20) H T AK = R, A = HRK 7 
where 



Here R,, is a k x k nonsingular triangular matrix. 

Note that by an appropriate choice of H and K of Eq. (3.20) we can 
arrange that A,, of Eq. (3.21) be either upper triangular or lower triangular. 

In summary, it has been shown that there are constructive procedures 
for producing orthogonal decompositions of the form A = HRK T in each 
of the six cases displayed in Fig. 1.1. In all the cases the rank k submatrix 
Ri , of Theorem (2.3) is obtained in triangular form. Thus it is particularly 
easy to compute the solution of Eq. (2.7). 

As already noted, in Cases la and 2a, the matrix KT of the decomposition 
can be taken to be the identity matrix /„. Likewise, in Case 3a, the matrix 
H can be taken to be the identity matrix I M . 

To illustrate these orthogonal decompositions we give a numerical exam¬ 
ple for each of the six cases of Fig. 1.1. 

(3.22) Case la Square, nonsingular 


m = n — 3, Rank (A) = 3 
QA — R 
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-0.4800 

-0.4129 

-0.7740] 

2 = 

0.6616 

-0.7498 

-0.0103 


-0.5761 

-0.5170 

0.6331. 


' 0.4087 

0.1593 

0.6593] 

A = 

0.3515 

0.9665 

0.6245 


0.6590 

0.9343 

0.9039. 


'—0.8514 

-1.1987 

-1.2740" 

R = 

0.0 

-0.6289 

-0.0414 


0.0 

0.0 

-0.1304 


(3.23) Case 2a Overdetermined, full rank 

m — 3, n = 2, Rank (j4) = 2 




-0.4744 

-0.4993 

-0.7250' 

2 = 

0.5840 

-0.7947 

0.1652 


.-0.6587 

-0.3450 

0.6687 


‘ 0.4087 

0.15941 


A = 

0.4302 

0.3516 



0.6246 

0.3384. 


R = 

“—0.8615 

L 00 

-0.4965“ 

-0.1304J 



(3.24) Case 3a Underdetermined, full rank 

m — 2, n — 3, Rank (A) = 2 


A = 

2 = 


4e = [R:0] 


" 0.4087 
0.1594 

'-0.4744 

-0.4993 

-0.7250 


0.4301 

0.3515 

0.5840 

-0.7947 

0.1652 


0,6246] 

0.3384J 

-0.6587" 

-0.3450 

0.6687 
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-0.8615 

-0.4965 


0.0 " 

-0.1304. 


(3.25) Case 1 b Square, singular 


m = n = 5, Rank (A) = 3 




r-0.0986 

—0.6926 

-0.5937 

-0.0522 

-0.3941“ 


0.3748 

0.0072 

-0.5424 

0.3529 

0.6639 

<2 = 

-0.1244 

0.2438 

-0.3484 

-0.8633 

0.2419 


0.2401 

-0.6653 

0.4813 

-0.2980 

0.4234 


0.8813 

0.1351 

-0.0160 

-0.1966 

—0.4076 


' 0.1376 

0.4086 

0.1594 

0.4390 

0.4113“ 


0.9665 

0.6246 

0.3383 

0.7221 

0.8746 

A = 

0.8285 

0.0661 

0.9112 

0.6266 

0.2327 


0.0728 

0.3485 

0.8560 

0.8348 

0.2474 


0.5500 

0.9198 

0.0080 

0.7610 

1.0506 


-0.9660 

0.0 

0.0 

0.0569 

-0.2520“ 


0.1279 

-0.6389 

0.0 

-0.4688 

-0.5963 


-0.0709 

0.1861 

-0.8383 

-0.4806 

0.1632 


0.0726 

-0.4109 

-0.5354 

0.7253 

-0.1144 


.-0.2002 

-0.6231 

0.1027 

-0.1411 

0.7357. 


‘ 1.4446 

1.6867 

1.2530' 



R = 

0.0 

-1.3389 

-0.1486 




0.0 

0.0 

1.1831 




(3.26) Case 2b Overdetermined, rank deficient 

m = 6, n — 5, Rank (A) = 3 
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"-0.0894 

-0.6279 

-0.5382 

-0.0473 

-0.3573 

-0.4221" 


0.3768 

0.0510 

-0.4986 

0.3522 

0.6813 

-0.1360 

0 = 

-0.1235 

0.2301 

-0.3633 

-0.8609 

0.2386 

0.0417 

0.3354 

-0.6906 

0.4092 

-0.2991 

0.3539 

0.1685 


0.7849 

0.1389 

-0.1992 

-0.1088 

-0.4751 

0.2956 


_—0.3259 

-0.2324 

-0.3500 

0.1768 

0.0153 

0.8281_ 


" 0.1376 

0.4087 

0.1593 

0.4308 

0.4163" 



0.9667 

0.6246 

0.3384 

0.8397 

0.8029 



0.8286 

0.0661 

0.9111 

0.7495 

0.1577 


A = 

0.0728 

0.3485 

0.8560 

0.8068 

0.2644 



0.5501 

0.9198 

0.0080 

0.7910 

1.0323 



0.6498 

0.2725 

0.3599 

0.5350 

0.3801_ 



—0.9846 

0.0 

0.0 

-0.0868 

-0.1520' 



0.1343 

-0.6392 

0.0 

-0.4230 

-0.6281 


K = 

0.0177 

0.1650 

-0.8460 

-0.4808 

0.1598 



-0.0467 

-0.3820 

-0.5248 

0.7488 

-0.1256 



.-0.1006 

-0.6468 

0.0942 

-0.1469 

0.7356 



' 1.5636 

1.7748 

1.4574' 




R = 

0.0 

-1.3537 

-0.1233 





0.0 

0.0 

1.1736. 





(3.27) Case 3b Underdetermined, rank deficient 




m = 4, n — 5, Rank (4) — 3 



1“—0.9757 

0.0103 

0.1390 

-0.1693' 


0.0 

-0.9989 

0.0298 

-0.0363 


0.0 

0.0 

-0.7729 

-0.6345 


.-0.2193 

-0.0458 

-0.6184 

0.7533. 


' 0.4087 

0.1593 

0.6594 

0.4302 

0.3516' 

0.6246 

0.3383 

0.6591 

0.9342 

0.9038 

0.0661 

0.9112 

0.6898 

0.1931 

0.1498 

0.2112 

0.8150 

0.7983 

0.3406 

0.2803. 
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'—0.4225 

0.0249 

0.3156 

-0.0047 

-0.8493' 


-0.1647 

-0.1529 

-0.9349 

-0.0530 

-0.2696 

K = 

-0.6816 

0.6377 

-0.0851 

0.1253 

0.3254 


-0.4447 

-0.4483 

0.1187 

-0.7222 

0.2561 


.-0.3634 

-0.6070 

0.0713 

0.6782 

0.1858. 


’ 0.9915 

0.0 

0.0 ‘ 



R = 

1.5246 

0.5840 

0.0 




1.2573 

-0.1388 

1.1077 




EXERCISES 

(3.28) Find an eigenvalue-eigenvector decomposition of the Householder 
matrix H = I — 2ww r , || w|| = 1. 

(3.29) Find an eigenvalue-eigenvector decomposition of the Givens reflec¬ 
tion matrix of Eq. (3.10). 

(3.30) Show that G of Eq. (3.10) is a Householder transformation matrix. 



4 ORTHOGONAL DECOMPOSITION 

BY SINGULAR VALUE DECOMPOSITION 


In this chapter we shall present another practically useful orthogonal 
decomposition of the m x n matrix A. In the previous chapter the matrix A 
was expressed as a product HRK T where R was a certain rectangular matrix 
whose nonzero elements belonged to a nonsingular triangular submatrix. We 
shall show here that this nonsingular submatrix of R can be further simplified 
so that it constitutes a nonsingular diagonal matrix. This decomposition is 
particularly useful in analyzing the effect of data errors as they influence 
solutions to Problem LS. 

This decomposition is closely related to the eigenvalue-eigenvector 
decomposition of the symmetric nonnegative definite matrices A T A and AA T . 
The standard facts concerning the eigenvalue-eigenvector decomposition of 
symmetric nonnegative definite matrices are summarized in Appendix A. 

(4.1) Theorem ( Singular Value Decomposition ) 

Let A be an m x n matrix of rank k. Then there is an m x m orthogonal 
matrix U, an n x n orthogonal matrix V, and an m x n diagonal matrix 
S such that 


(4.2) U T A V=S, A = USV T 

Here the diagonal entries of S can be arranged to be nonincreasing; all 
these entries are nonnegative and exactly k of them are strictly positive. 

The diagonal entries of S are called the singular values of A. It will be 
convenient to give the proof of Theorem (4.1) first for the special case in 
which m = n — Rank (/I). The more general statement follows easily from 
this special case. 


18 
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(4.3) Lemma 

Let A be an n x n matrix of rank n. Then there exists an n x n 
orthogonal matrix U, an n x n orthogonal matrix V, and an n x n 
diagonal matrix S such that 

(4.4) U T AV= S, A = USV 1 " 

The successive diagonal entries of S are positive and nonincreasing. 

Proof ofLemma (4.3): The positive definite symmetric matrix A T A has 
an eigenvalue-eigenvector decomposition 

(4.5) A T A = VDV 7 

where the n x n matrix V is orthogonal and the matrix D is diagonal with 
positive nonincreasing diagonal entries. 

Define the diagonal matrix S to be the n x n matrix whose diagonal 
terms are the positive square roots of the respective diagonal entries of D. 
Thus 

(4.6) D = S t S= S 1 
and 

(4.7) S~ t DS~ l = l„ 

Define the n x n matrix 

(4.8) l/ = ^KS-‘ 

From Eq. (4.5), (4.7), and (4.8) and the orthogonality of V, 

(4.9) U T U = S 1 V T A T A VS' 1 

= S'DS -> 

= L 

so that U is orthogonal. 

From Eq. (4.8) and the fact that V is orthogonal, 

(4.10) USV 1, = AVS~ X SV T 

= AVV T 
= A 

This completes the proof of Lemma (4.3). 
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Proof of Theorem {4.1): Let 


(4.11) 


A — HRK? 


where H, R, and AT have the properties noted in Theorem (3.19). 

Since the k x k matrix R tl of Eq. (3.21) is nonsingular, Lemma (4.3) 
shows that we may write 


(4.12) 


R u = tfSP 7 


Here 0 and P are k x k orthogonal matrices, while S is a nonsingular diago¬ 
nal matrix with positive and nonincreasing diagonal elements. 

It follows from Eq. (4.12) that the matrix R of Eq. (3.21) may be written as 


(4.13) 


R = 0S? T 


where 0 is the m x m orthogonal matrix 


(4.14) 


-[o LI 


9 is the n x n orthogonal matrix 


(4.15) 


-rj 


and S is the m x n diagonal matrix 


(4.16) 

Then defining U and V as 

(4.17) 

(4.18) 


_rS or 

~|o 0 


u=>h0 


we see from Eqs. (4.11H4.18) that 


(4.19) A = USV 7 

where U, S, and Fhave the properties stated in Theorem (4.1). This completes 
the proof of Theorem (4.1). 

Notice that the singular values of the matrix A are uniquely determined 
even though the orthogonal matrices U and V of Eq. (4.19) are not unique. 
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Let a be a singular value of A with multiplicity /. Thus among the ordered 
singular values there is an index i such that 

o = s, for/ = /,/ + 1./ + /- 1 

and 

a ^Sj for j < i otj ;> / + / 

The /-dimensional subspace T of n-space spanned by the column vectors v Jt 
j — i,i+ 1— 1, of Pis uniquely determined by A but the specific 
set of orthogonal basis vectors of T constituting columns i, 
i + / — 1 of V is not uniquely determined. 

More specifically, let k = min (m, ») and let Q be a k x k orthogonal 
matrix of the form 



0 

0 


where Pis lx l and orthogonal. If A — USV T is a singular value decomposi¬ 
tion of A and s, = j, + , = • • • = then Ost* with 




andT 


= v\° 1 

Lo 'r.J 


is also a singular value decomposition of A. 

The foUowing numerical example gives the singular value decomposition 
of the matrix A introduced previously as Example (3.23) for Case 2a. 


(4.20) 



S = 

V = 


A = 

®03 rr 

-0.4347 

0.6141 -0.6587' 

-0.5509 

-0.7599 -0.3450 

-0.7125 

0.2129 0.6687. 

r 0.9965 

0.0 I 

L 0.0 

0.1128J 

r—0.8626 

0.50581 

L—0.5058 

—0.8626J 
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EXERCISES 

(4.21) If A is n x n and symmetric with distinct singular values, then 

(a) A has distinct (real) eigenvalues. 

(b) A singular value decomposition of A can be used to obtain 
an eigenvalue-eigenvector decomposition of A and conversely. 

What if the singular values are not distinct? 

(4.22) If 5, is the largest singular value of A, then || A || = s ,. 

(4.23) If R is an n x n nonsingular matrix and s , is its smallest singular 
value, then 1| = s’ 1 . 

(4.24) Let Si and s„ respectively, denote the largest and smallest singular 
values of a matrix A m „ of rank n. Show that Jj|x|| <; |Mjc|| ^ 
Stjlxll for all n-vectors x. 

(4.25) Let s u .,., s k be the nonzero singular values of A. Then 

(4.26) Let A ~ USV T be a singular value decomposition of A. Show that 
the column vectors of U are eigenvectors of the symmetric matrix 
AA T . 



5 PERTURBATION THEOREMS 
FOR SINGULAR VALUES 


The singular values of a matrix are very stable with respect to changes 
in the elements of the matrix. Perturbations of the elements of a matrix 
produce perturbations of the same, or smaller, magnitude in the singular 
values. The purpose of this chapter is to present Theorems (5.7) and (5.10), 
which provide precise statements about this stability, and Theorem (5.12), 
which gives bounds on the perturbation of singular values due to removing 
a column, or row, of a matrix. 

These theorems are direct consequences of corresponding theorems about 
the stability of eigenvalues of a symmetric matrix. We first quote the three 
relevant eigenvalue theorems. 

(5.1) Theorem 

Let B, A, and E be n x n symmetric matrices with B — A = E. 
Denote their respective eigenvalues by fi it a„ and i = 1,..., n, each 
set labeled in nonincreasing order. Then 

c.<. fi,~ *,<,(i i = 1 ,....n 

A weaker conclusion that is often useful because it requires less detailed 
information about E is 

IA — #<l <, roaxjfyl = ||£|| i=l. n 

(5.2) Theorem ( Wielandt-Hoffman) 

With the same hypotheses as in Theorem (5.1), 

g (fit - «,) 2 eh = ||£||) 


23 



24 PERTURBATION THEOREMS FOR SINGULAR VALUES 


CHAP. 5 


(S.3) Theorem 

Let A be an n X n symmetric matrix with eigenvalues «,;>«*;>•■• 
> Le/ k be an integer, 1 <, k <, n. Let B be the (n — 1) x (n — 1) 
symmetric matrix obtained by deleting the k th row and column from A. 
Then the ordered eigenvalues of B interlace with those of A. as follows: 


The reader is referred to Wilkinson (1965a), pp. 99-109, for discussion 
and proofs of these three theorems as well as the Courant-Fischer Minmax 
Theorem on which Theorems (5.1) and (5.3) depend. Theorem (5.2) is due to 
Hoffman and Wielandt (1953) and is further treated in Wilkinson (1970). 

To derive singular value perturbation theorems from these eigenvalue 
perturbation theorems, we shall make use of the relationship between the 
singular value decomposition of a matrix A and the eigenvalue-eigenvector 
decomposition of the symmetric matrix 



If A is square with a singular value decomposition A — USV T , it is easily 
verified that C has an eigenvalue-eigenvector decomposition 


(5.5) 

where 


and 



U = 2~ i/i U 


p= 2- x/i V 


If d mXM , m^>n, has a singular value decomposition 



then C, as defined in Eq. (5.4), has the eigenvalue-eigenvector decomposition 



S 0 0 
C = P 0 -S 0 
0 0 0 


(m-ii) x (m-n) J 
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where 


and 


— OH) lfU> "I 
fjw — 2-utijiu 


^ = 2~ ,/a K 


Clearly, analogous results hold for the case of m < n. For later reference 
we state the following theorem based on the discussion above: 

(5.6) Theorem 

Let Abe mm x n matrix and k = min (m, n). Let C be the (m + n) X 
(m + n) symmetric matrix defined by Eq. (5.4). If the singular values of 
A are s„ ..., s„, then the eigenvalues of C are s,,..., s k , — s t ,...» 
—s k , and zero repeated | m — n | times. 

We may now state the following three theorems regarding the perturba¬ 
tion of singular values. 

(5.7) Theorem 

Let B, A, and E be m x n matrices with B — A — E. Denote their 
respective singular values by fi t , a„ and e„ i = 1,..., k; k = min 
(m, n), each set labeled in nonincreasing order. Then 



=l|£|l 



k 


Proof: Introduce the three symmetric matrices 


(5.9) 

Then 




*0 A' 
A T 0 


and 



B-A = E 


The eigenvalues of these matrices are related to the singular values of B, A, 
and E as stated in Theorem (5.6). Applying Theorem (5.1) to the matrices 
S, A, and £, we obtain the result in Eq. (5.8). 


(5.10) Theorem 

With hypotheses as in Theorem (5.7), there follows the inequality 




(5.11) 
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Proof: Introducing Eq. (S.9) and using Theorems (S.6) and (5.2), we 

obtain 

which is equivalent to Eq. (5.11). 

(5.12) Theorem 

Let A be an m x n matrix. Let k be an integer , 1 <; k n. Let B 
be the m x (n — 1) matrix resulting from the deletion of column k 
from A. Then the ordered singular values fl l of B interlace with those 
0C| of A as follows: 

Case Im^d 

(5.13) 

Case 2 m < n 

(5.14) <*»2: A.2>0 

Proof: The results in Eq. (5.13) and (5.14) follow directly from appli¬ 
cation of Theorem (5.3) to the symmetric matrices A = A r A and & = B r B. 

In Case 1 the eigenvalues of A and & are a?, i = 1. n, and fl}, i — 

1— 1, respectively. In Case 2 the eigenvalues of A are a}, i = 1,..., 
m, and zero repeated n — m times, while the eigenvalues of S are fif, i — 
1,..., m, and zero repeated n — 1 — m times. 


EXERCISES 

(5.15) Problem [Eckart and Young (1936)]: Given an m x n matrix A, of 
rank k, and a nonnegative integer r < k, find an m x n matrix B of 
rank r that minimizes \\B — A\\ r . 

Solution: Let A = USV T be a singular value decomposition of A 
with ordered singular values j, ;> s 2 ^ s k > 0. Let 5 be 
constructed from S by replacing $,+, through s k by zeros. Show 
that S = USV* solves the stated problem, and give an expression for 
|| B — A II, in terms of the singular values of A. 

Remark: The proof is straightforward using Theorem (5.10). The 
proof given by Eckart and Young contains the essential ideas needed 
to prove Theorem (5.10). 

(5.16) Solve the problem of Exercise (5.15) with || • |[f replaced by || • ||. 
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(5.17) Define k(A) to be the ratio of the largest singular value of A to the 
smallest nonzero singular value of A. (This w condition number” of 
A will be used in subsequent chapters.) Show that if Rank (v4 -x J — n 
and if B mxr is a matrix obtained by deleting (n — r) columns from 
A, then k(B) <, k(A). 



6 BOUNDS FOR THE CONDITION NUMBER 
OF A TRIANGULAR MATRIX 


In the discussion of the practical solution of least squares problems in 
Chapter 25, the need will arise to estimate the largest and smallest nonzero 
singular values of a matrix A since the ratio of these two quantities (the con¬ 
dition number of A) has an interpretation of being an error magnification 
factor. This interpretation of the condition number will be presented in 
Chapter 9. 

The most direct approach is to compute the singular values of A (see 
Chapters 4 and 18 and Appendix C). One may, however, wish to obtain 
bounds on the condition number without computing the singular values. 
In this chapter we present some theorems and examples relating to this 
problem. 

Most of the algorithms which are described in this book produce as 
an intermediate step a nonsingular triangular matrix, say, R, which has 
the same nonzero singular values as the original matrix A. In general, a non¬ 
singular triangular matrix is a better starting point for the bounding of singu¬ 
lar values than is a full matrix. Therefore we shall deal only with bounds for 
the singular values of a nonsingular triangular matrix R. 

Denote the ordered singular values of the n x n nonsingular triangular 
matrix R by s , ;> .r 2 > ■ • • > s m > 0. As noted in Exercise (4.22), j, = || R ||. 
Therefore, one easily computable lower bound for j, is 

(6.1) j, > max, y | r,j | 

Furthermore since R is triangular, the reciprocals of the diagonal elements 
of R are elements of R~\ Therefore 

(6.2) li/?‘ , ll>max / |r < 7 , | 
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From Exercise (4.23), s~ l = j| J?‘‘ ||, and thus 
(6.3) min, | i-al 


Therefore, a lower bound p for the condition number x = sjs m is avail 
able as 


(6.4) 



This lower bound p, while of some practical use, cannot in general be 
regarded as a reliable estimate of x. In fact, x can be substantially larger 
than p. 

As our first example [Kahan (1966b), p. 790], let the upper triangular 


n x n matrix R be defined by 




0 

if j < i 

II 

*** 

c 

sr 

i 

it/-/ 


-i 

if j > i 

For example, for n — 4 we have 



n 

-i 

-1 - 


( 6 . 6 ) 


R = 


0 1 - 1-1 
0 0 1-1 

0 0 0 1 


Using Eq. (6.4) we find p— 1 as a lower bound for x, which tells us nothing 
at all. For this matrix a more realistic upper bound for s m can be obtained by 
noting the effect of R on the vector y with components 

(6.7) y, = 2 W for i — 1. n — 1 

y. = y .-1 

It is easily verified that Ry = z where 

(6.8) z, — 0 for i = 1. n — 1 

z, = 2 1 '* 


Then, using the inequalities 
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we obtain 

( 6 . 10 ) k = ^ 2 "-* 


The inequality of Eq. (6.9) may also be interpreted as showing that R is 
close to a singular matrix in the sense that there exists a matrix E such that 
II £II <£ 2*"* and R — E is singular. It is easily verified that such a matrix is 
given by E = zy r /y r y. 

To illustrate this example for the case of n = 4, we note that y — 
(1, ±Y, z = (0,0,0, ^) r , j« <; and k > 4. Furthermore, subtracting 

the matrix 


' 0 0 0 0 * 
0 0 0 0 
0 0 0 0 


Li 2 r tV Jij 


from the matrix R of Eq. (6.6) gives a singular matrix. 

This example illustrates the possible danger of assuming a matrix is well 
conditioned even when the p of Eq. (6.4) is small and the matrix is “innocent 
looking.” 

Recall that we are primarily interested in triangular matrices that arise 
from algorithms for solving systems of linear equations. The matrix defined 
by Eq. (6.S) has been discussed in the literature because it is invariant under 
Gaussian elimination with either full or partial pivoting. It is not invariant, 
however, under elimination by Householder transformations using column 
interchanges (such as the Algorithm HFTI, which will be described in Chap¬ 
ter 14). 

As a numerical experiment in this connection we applied Algorithm 
HFTI to the matrix R of Eq. (6.5). Let R denote the triangular matrix 
resulting from this operation. We also computed the singular values of R. 
The computing was done on a UNIVAC 1108 with mixed precision (see 
Chapter 17) characterized by single and double precision arithmetic, respec¬ 
tively, of 


if = 2~* T = 0.745 x 10-* and co = 2 s * = 0.347 x 10' ,T 

Table 6.1 lists the computed values of the last diagonal element and 
the smallest singular value s n as well as the ratio sjf„ for n = 20 through 26. 
Observe that in these cases provides a good estimate of the size of s„. 

It is nevertheless true that there exist n x n matrices with unit column 
norms which could be produced by Algorithm HFTI and whose smallest 
singular value is smaller than the smallest diagonal element by a factor of 
approximately 2 1 
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Tabic 6.1 LAST DIAGONAL ELEMENT OF R AND LAST SINGULAR VALUE OF X 


n 

f*. x 10* 

S m X 10* 

Sjfm 

20 

330. 

286. 

0.867 

21 

165. 

143. 

0.867 

22 

82.5 

71.6 

0.868 

23 

40.5 

35.8 

0.884 

24 

20.5 

17.9 

0.873 

25 

8.96 

8.95 

0.999 

26 

4.59 

4.52 

0.985 


The following example of such a matrix is due to Kahan (1966b), pp. 791— 
792. Define the n x n matrix R by 



if i>j 
if/—y 

if i < j 


where s and c are positive numbers satisfying s 2 + c* = 1. For example, if 
n — 4, we have 


( 6 . 11 ) 



—c — c 
—cs —cs 
s* — cs 2 
0 s 1 


For general n, the matrix R is upper triangular, the column vectors are all of 
unit euclidean length, and the inequalities 

(612) rl k ^ ^ r?j for k = 1. n - 1 

and j = k + 

are satisfied. As established in Chapter 14, these inequalities imply that R 
is a matrix that could result from applying the Algorithm HFTI to some 
matrix A. 

Let T = R - 1 . The elements of T are given by 


0 

if j < i 

i 

jF 7 

if j = i 

c 

s'-' 

if y = i+l 

c(i + cy-‘- 1 

if 7 -i^2 

3*~ l 
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For example, if n => 4, we have 



Let t„ denote the last column vector of R~'. Then 

„ T Mi _ 1 + c*[( 1 + e)*-* - l] r _i 

As s —► 0 + and c = (1 — j 1 ) 171 —»1", the product ||T,||*ri approaches 
(4'* -1 + 2)/3 from below. Thus there is some value of s, depending on n, for 
which 

llr.H 

Then, using. Exercise (4.24), 



we have 

s.^3‘ 7 *2‘-|r M | = (1.73)2>-|r„| 

The following theorem shows that this case represents nearly the minimum 
value attainable by sj\ r„ |. 

(6.13) Theorem [Stated without proof in Faddeev, et al. (1968)] 

Let A be an m X n matrix of rank n. Assume all column vectors of A 
have unit euclidean length. Let R be an upper triangular matrix pro¬ 
duced by Householder triangularization of A with interchange of 
columns as in Algorithm HFTI (14.9). Then s a , the smallest singular 
value of A, is related to r aa , the last diagonal element of R, by 

(614) s„ <. | r m [ 

and 


(6.15) 


s m ^ 3(4" -f 6 n- l)-‘ /J |r aa | ^ 2»~" IrJ 
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Proof: From the way in which R is constructed, its column vectors 
have unit euclidean length and it has the same singular values as A. Further¬ 
more, as a result of the use of column interchanges, the following inequalities 
will hold: 

(6.16) for k = 1. n - 1 

and j = k + 1. n 

From Eq. (6.16) it follows that 

(6.17) |r**|^|r M | for k, j>k,k= 1.n-1 

Since 

(6.18) = 

the assertions in Eq. (6.14) and (6.1S) are equivalent to the assertions 

(6-19) ll*- 1 II SSI'S. 1 1 

and 

(6.20) ||*-‘ || <; 3-‘(4« + 6« - 1)»'* |r-» | 

The assertion in Eq. (6.19) follows from the fact that r~ m l is an element of 
R-*. 

To establish Eq. (6.20), we shall develop upper bounds on the magnitudes 
of all elements of R~ f and then compute the Frobenious norm of the bound¬ 
ing matrix J if. 

Define T — R~K It will be convenient to group the elements of T into 
diagonals parallel to the main diagonal. Introduce bounding parameters 
for the diagonals as follows: 

(6.21) g* = max (|/,.,♦* |, i = l,...,n — k] for k = 0, 1, ... ,n - 1 
Then 


(6.22) g 0 = max, |f„| = max, |r,7* | = |r"'| 


The elements in the fcth superdiagonal of T are expressible in terms of ele¬ 
ments of the main diagonal and preceding superdiagonals of T and elements 
of 21 as follows: 


(6.23) 



1 <£/£«- 1 
1 <£ k ^ n — / 
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Since | r, Ml | <, | r„ |, we may use Eq. (6.21) and (6.23) to obtain 


(6.24) 



= max, | t (Mk | 
^max,|j|f, +M+ * 



It is then easily verified by induction that 


(6.25) g„ <; 2= 2*-* |r-‘ | for* = 1, 


Define an n x n upper triangular matrix AS by 

(0 if j < i 

(6.26) nifj = 11 ify -1 

[V-'-' if j > i 

and define 


(6.27) 

For example, if n — 4, 





'112 4' 
0 112 
0 0 11 
0 0 0 1 


n 


From Eq. (6.21) and (6.25) to (6.27), it follows that the elements of R~* 
(= T) are individually bounded in magnitude by the corresponding elements 
of M. Thus 

(6.28) *-« - ||H-‘|i £ Hit-II, <£ IIASribr = |r-> \.\\M\\, 


To compute the Frobenious norm of AS, we note that for y ^ 2 the sum of 
squares of the elements of column j is 


(6.29) 




4'-‘ +2 
3 


This expression is also correct for j = 1, assuming the sum in the middle 
member is defined as zero. 

Then 

(6.30) ||*-‘ HI = || TUI <£ || ASUI - i- t M-? = rj£±*j^± 
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The inequalities in Eq. (6.28) and (6.30) together establish the inequality 
in Eq. (6.20) and complete the proof of Theorem (6.13). 

The following more general theorem has also been stated by Faddeev, 
Kublanovskaya, and Faddeeva (1968). 

(6.31) Theorem 

Let A and R be matrices as given in Theorem (6.13). The singular 
values s,> s 3 ;> • • ■ > s, of A are related to the diagonal elements 
r„ of R by the inequalities 

(6.32) 2“'|r w |^3(4' + 6r - 1) ' ■'* \r u \ £ s, <. (n - i + l) 1 ' 1 1r„| 

for i = 1 

Proof: Define Rj to be the leading principal submatrix of R of order 
j. Denote the *th singular value of R, by s\ n . Using Theorem (5.12) we have 
the inequalities 

(6.33) s, = j/-> ^ ^ ^ s>'> 

Theorem (6.13) may be applied to R, to obtain 

(6.34) s, v> ;> 3(4‘ + 6/ — \)~ m \r„\ 

which with Eq. (6.33) establishes the lower bound for s, stated in Eq. (6.32). 

Define W, to be the principal submatrix of R of order n + 1 — j consist¬ 
ing of the intersections of rows and columns j through n. Note that the 
leading diagonal element of W t is r„. Using the properties in Eq. (6.16) and 
Exercise (6.37), one obtains 

(6.35) l|H'iiK(«+ 1 -0 ,/a |r M | 

Denote the ith singular value of W } by a>} J) . Then from Theorem (5.12) 
we may write 

(6.36) s, — m} n ^ co}l\ < - <,mV 

Since = || W t |j, Eq. (6.35) and (6.36) may be combined to obtain the 
upper bound for s, in Eq. (6.32), completing the proof of Theorem (6.31). 

EXERCISE 


(6.37) Let A be an m x n matrix with column vectors a } . Then [| A\\ <, 
n ui maxjWajW. 
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If A is an n x n nonsingular matrix, the solution of the problem Ax — b 
can be written as x = A~'b where A~ x is the (unique) inverse matrix for A. 
The inverse matrix is a very useful mathematical concept even though effici¬ 
ent and reliable contemporary methods [Forsythe and Moler (1967); Wilkin¬ 
son (196Sa)] for computing the solution of Ax — b do not involve explicit 
computation of A~ l . 

For Problem LS there arises the question as to whether there exists some 
n x m matrix Z, uniquely determined by A, such that the (unique) minimum 
length solution of Problem LS is given by x = Zb. This is indeed the case 
and this matrix Z is called the pseudoinverse of A. As noted above for the 
inverse, the pseudoinverse is a useful mathematical concept but one would 
not usually compute the pseudoinverse explicitly in the process of computing 
a solution of Problem LS. 

The following two theorems lead to a constructive definition of the pseu¬ 
doinverse of an m x n matrix A. 

(7.1) Theorem 

Let A be an m x n matrix of rank k with an orthogonal decomposition 
A = HRK T as in the hypotheses of Theorem (2.3). Then the unique 
minimum length solution of Problem LS is given by 

(7.2) * - *(*“ “)™ 

Proof: The conclusion, Eq. (7.2), is an alternative way of writing Eq. 
(2.5) to (2.7) and Eq. (2.11). 
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(7.3) Theorem 

Let A,*, = HRK T as in Theorem (7.1). Define 

z - y 

Then Z is uniquely defined by A; it does not depend on the particular 
orthogonal decomposition of A. 

Proof: For each j, 1 <; j <, m, thejth column 2 } of Z can be written as 
Zj — Ze„ where e, is the /th column of the identity matrix I m . From Theorem 
(7.1), Zj is the unique minimum length solution of the least squares problem 
Ax, S e t . This completes the proof of Theorem (7.3). 

In view of Theorems (7.1) and (7.3) we make the following definition. 

(7.4) Definition 

For a general m x n matrix A, the pseudoinverse of A, denoted by 
A*, is the n x m matrix whose y'th column z, is the unique minimum 
length solution of the least squares problem 

Ax, S e, 

where e, is theyth column of the identity matrix /„. 

This definition along with Theorems (7.1) and (7.3) immediately allow us 
to write the minimum length solution to Problem LS as 

(7.5) x = A*b 

The following two cases are worthy of special note. For a square nonsin¬ 
gular matrix B, the pseudoinverse of B is the inverse of B: 


(7.6) 

For an m x n matrix, 



with Ri, a k x k nonsingular matrix, the pseudoinvetse of It is the n x m 
matrix 


( 7 . 7 ) 


- - {? 0 
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The pseudoinverse of an m x n matrix A can be characterized in other 
ways. Each of the following two theorems provides an alternative character¬ 
ization of the pseudoinverse. 

(7.8) Theorem 

If A — HRK T is my orthogonal decomposition of A, as given in 
Theorem (2.3), then A + = KR + H T where R + is given in Eq. (7.7). 

(7.9) Theorem [The Penrose Conditions; Penrose (7955)) 

The n x m matrix A + is the unique matrix X that satisfies the following 
four conditions: 

(a) AXA = A 

(b) XAX = X 

(c) (AX) T = AX 

(d) (XA) T - XA 

Proofs of these two theorems are left as exercises. 

The explicit representation of A* given in Theorem (7.8) is particularly 
useful for computation. For example, if one has the orthogonal decom¬ 
position A of Eq. (3.20), with R ,, of Eq. (3.21) nonsingular and triangular, 
then 

(7.10) A* = 

If one has the singular value decomposition of Eq. (4.2), as given in Theo¬ 
rem (4.1), then 

(7.11) A* = VS + U T 

As mentioned at the beginning of this chapter, there is usually no need 
to construct the pseudoinverse of a matrix A explicitly. For example, if the 
purpose of the computation is the solution of Problem LS, then it is more 
economical in computer time and storage to compute the solution with the 
following three steps [which are essentially repeated from the proof of Theo¬ 
rem (2.3)]. 


(7.12) 

(7.13) Solve R tt y, = g, fory,. 


g = H T b 
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EXERCISES 

(7.15) Verify Eq. (7.6) and (7.7). 

(7.16) Prove Theorem 7.8. 

(7.17) [Penrose (1955)] Prove Theorem 7.9. 

(7.18) Prove (A*f = (A*)* and (/l + ) + = A. 

(7.19) Prove: If Q mxm has orthonormal columns or orthonormal rows, then 

If Q mxm is of rank n and satisfies Q* = Q r , then Q has or¬ 
thonormal columns. 

(7.20) [Penrose (1955)] If U and V have orthonormal columns, then {UA K T ) + 
= VA*U T . 

(7.21) (a) Let A = USV T be a singular value decomposition of A. Write 

singular value decompositions for the four matrices P, = A*A, 
P 2 = / — A* A, Pj = A A*, and P« = / — A A* in terms of the 
matrices U, S, and V. 

(b) Deduce from part (a) that the matrices P, are symmetric and idem- 
potent, hence projection matrices. 

(c) Let T, denote the subspace associated with the projection matrix 
P,; that is, T, — [x: P,x — x). Identify the relationship of each 
subspace T, to the row or column spaces of A. 

(7.22) Prove that the most general solution of Ax S b is x =* A + b + (/ — 
A*A)y, where y is arbitrary. 

(7.23) [Greville (1966)] For nonsingular matrices one has the identity(/tJ9)~ 1 
= B~ X A~*. The analogous equation (ABy = B*A* is not satisfied 
by all matrices A MXk and B kxm . 

(a) Exhibit two matrices A and B with m — n = I and k — 2 such 
that (ABy ^ B*A*. 

(b) Prove that matrices A mxk and B kXH satisfy {ABy — B+A* if and 
only if the range space of B is an invariant space of A r A and the 
range space of A T (row space of A) is an invariant space of BB T . 

(7.24) If Rank (v4„ x .) = n, then A* = {A T A)-'A T . 

If Rank (A mx J = m, then A + — A^AA 1 ) -1 . 

(7.25) [Penrose (19S5); Graybill, et al. (1966)] Given A, the equations XAA r 
— A r and A T A Y — A r are each consistent. If matrices X and Y 
satisfy these two equations, respectively, then XA and A Y are projec¬ 
tion matrices (symmetric and idempotent) and A* = XAY. Note the 
simplification of this proposition in case A is symmetric. 
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(7.26) [Graybill, et al. (1966)] The pseudoinverse of a rectangular matrix A 
can be defined in terms of the pseudoinverse of the symmetric matrix 
A T A or AA r , whichever is more convenient, by the formulas A* — 
(A T A) + A T or A* — A^AA 7 )*, respectively. 

(7.27) [Penrose (1955)] If A is normal (i.e., satisfies A T A = AA 7 }, then A*A 
= AA* and (A-)* = (A+y. 

(7.28) [Penrose (1955)] If A = 'LA I with A,A T , = 0 and AJA, = 0 whenever 
i *j, then A* = Z At. 



8 PERTURBATION BOUNDS 
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Our objective here and in Chapter 9 is to study the relationship of per¬ 
turbation of the data of Problem LS to perturbation of the solution of the 
problem. In this chapter we develop the perturbation theorems for the pseudo¬ 
inverse. These theorems are then used in Chapter 9 to study perturbation of 
the Problem LS. 

In practice the consideration of such perturbations can arise due to the 
limited precision with which observable phenomena can be quantised. It is 
also possible to analyze the effects of round-off errors in the solution proce¬ 
dure as though their effects were due to perturbed data. This analysis for 
algorithms using Householder transformations will be described in Chapters 
15,16, and 17. 

Results relating to perturbation of the pseudoinverse or the solution of 
Problem LS have been given by a number of authors. The treatment of this 
problem by Wedin (1969) seems most appropriate for our present purposes in 
terms of generality and the convenient form of the final results. For earlier 
treatments of this perturbation problem or special cases of the problem, see 
Golub and Wilkinson (1966), Bjdrck (1967a and 1967b), Pereyra (1968), 
Stewart (1969), and Hanson and Lawson (1969). 

Let A and E be m x n matrices and define the perturbed matrix 

(8.1) X = A + E 
and the residual matrix 

(8.2) G = X* — A* 

We wish to determine the dependence of G on fand in particular to obtain 
bounds for || G|1 in terms of \\A || and || E||. 


41 
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It will be convenient to introduce the four projection matrices: 

(8.3) P = A* A = A T A T+ , Q = AA + = A T *A T 

(8.4) P = A* A = A*A”, Q = AA* = A”A' 


These matrices have many useful properties derivable directly from the 
Penrose conditions [Theorem (7.9)]. Also see Exercise (7.21) and the sum¬ 
mary of standard properties of projection matrices given in Appendix A. 

The matrices defined in Eq. (8.1) to (8.4) will be used throughout this 
chapter without further reference. 

(8.5) Theorem 

Using definitions of Eq. (8.1) and (8.2), the matrix G satisfies 


( 8 . 6 ) 

where 

(8.7) 

( 8 . 8 ) 

(8.9) 


G = G x + Gi + <?, 

G, = -Z+EA+ 

G 2 - A*(I - Q) = A*A r *E T (I - Q) 
G 3 = -(/ - P)A* - (/ - i s )£ r ^ T+ ^ + 


These matrices are bounded as follows: 


( 8 . 10 ) 

(8.11) l|Gill£ll*IMtf’ + IP 

(8.12) ll<?,ll£ll*ll-M*IP 


Proof: Write G as a sum of eight matrices as follows: 

(8.13) G = (/* + (/ — P))-(A+ - A+)-fQ + (/ - 0] 

- PA+Q + PA*(I- Q) - PA+Q - PA*(I- Q) 

+ (/-/ 5 )^ + e + (/- p)i + (/ - 0 
- a - /v + e - a - P)A + (i - 0 


Using the properties 

PA* = (/ - /*>?+ = 0 

A+Q = A+, A*(I -0 = 0 

Eq. (8.13) reduces to 

G = (1 + G - /U + ) + ^ + (/ - 0 - (/ - i*M + 

= C, + Gl + Gj 



CHAP. 8 


perturbation bounds for the pseudoinverse 43 


To expose the first-order dependence of G on E, write 

(8.14) G, = 1 + AA + - A + Aa + = -A+EA+ 

G 2 = A + Q(I - Q) = A*A T *A T {I - Q) 

= A*A T *(A T - A T )(I - Q) = A + A T *E T (I - Q ) 

G, = -(/ - = -(/ - P^FjF*A* 

= -(/ - /%4 r - A t )A t+ A* = (/ - ^)£T/t r ^ + 

The bounds in Eq. (8.10) to (8.12) follow from the facts that ||/ — <2|| < 1 
and \\I — I’ll <, 1, completing the proof of Theorem (8.5). 

We remark that for the case of real numbers a and & (and in fact for square 
nonsingular matrices) one has the algebraic identity 




which suggests that Eq. (8.10) is a reasonable form to expect in a bound for 
||G||. The additional terms G t and G, are specifically associated with the 
cases of nonsquare matrices or square singular matrices in the following 
sense: The matrix G 2 can be nonzero only if Rank (_A) < m and G 3 can be 
nonzero only if Rank (A) < n since Q — I m if Rank (A) — m and P—I n \f 
Rank (A) = n. 

We next wish to replace ||^ + || in the right sides of Eq. (8.10), (8.11) and 
(8.12) by its bound in terms of || || and ||£||. Such a bound is available 

only under the assumptions in Eq. (8.16) and (8.17) of the following theorem. 

(8.15) Theorem 
Assume 

(8.16) Rank (A+ E)<, Rank (A) = k> 1 
and 

(8.17) il'**IHI£|l < 1 


Let Sk denote the smallest nonzero singular value of A and f = || E||. 
Then 

(8.18) Rank {A + E) = k 

and 


(8.19) 


1104 + £)♦!!<£ 



l^ + 

1. 4 

1-1 

A+\ 

[TO 




e 


Proof: The inequality of Eq. (8.17) can be written as e/s k < 1 or 
equivalently s k — e > 0. Let S k denote the kth singular value of A == A + E. 
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From Theorem (5.7) 

( 8 . 20 ) I k ^s k - ( 

which implies that Rank (A + E) >: k. With the inequality of Eq. (8.16) 
this establishes Eq. (8.18). The inequality of Eq. (8.20) can be written as 


which is equivalent to the inequality of Eq. (8.19). This completes the proof 
of Theorem (8.15). 

The conditions of Eq. (8.16) and (8.17) are necessary. It is easy to verify 
that \\A* || may be unbounded if either of these conditions are not satisfied. 
As long as we shall be imposing Eq. (8.16) and (8.17) to obtain Eq. (8.19), we 
can take advantage of the condition in Eq. (8.18) to prove that ||£||*||iJ + 1|* 
can be replaced by ||£t[«||i4'*’||<||/7 + [| in the bound for ||<7 2 || given in Eq. 
(8.11). This is accomplished by Theorems (8.21) and (8.22). 

(8.21) Theorem 

If Rank (A) = Rank (A), then 

II5(/-G)IIHIG(/-0)II 


Proof: This proof is due to F. T. Krogh. Write singular value 
decompositions 

A = USV T 


and 


A = QSP 7 


Then from Eq. (8.3) and (8.4) and the assumption that A and A have the 
same rank, say, k, 

e - t, [o oh 

and 



Define the m x m orthogonal matrix W with submatrices W tJ by 
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Then 

iifi( '- o)i HKo »Ho /.°>ll 

-K K ?]|-«- 

Similarly, one cart verify that 

II G(/- 6)11 -II^bi II 


It remains to be shown that |j W l3 || = || W tl ||. Let x be any (m — /c)-vector 
and define 



Then, using the orthogonality of W, 


I 1 * 11 * - 1 WP - II w 

Hl»r.l*ll 1 +ll»'»**ll a 

and thus 

W„x\\* = 11**11- II W„x\\* 

Therefore 

ll^iall 1 - max||W' ll *||» = 1 - min||W'„x||* 
11 * 11-1 11 * 11-1 

= 1 - s* m . k 


where is the smallest singular value of W u . 

Similarly, from 

11*11* = llyll* - II ^ll* - II»T.jc|P + ll»7i*ll* 


one obtains 

|| W u ||* = 1 - min || HT,jc||* - 1 - **.* 

ii* ii-1 

Thus || W'nll = \\W 2l \\, which completes the proof of Theorem (8.21). 
(8.22) Theorem 

If Rank (A) = Rank (A), then the matrix G 2 defined in Eq. (8.8) satis¬ 
fies 


(8.23) 


IIG*H£»M + IHI* + II 
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Proof: Using Rank (A) — Rank (A), Theorem (8.21) permits us to 

write 

IIC.II^II^IHIfia-011 

Then since 

W-Q) = A*+A r (I - Q) 

= A r *(A T - - Q) = —A T *E T (I - Q) 

it follows that [| || satisfies Eq. (8.23), which completes the proof of Theo¬ 

rem (8.22). 


We are now in a position to establish the following theorem, which may 
be regarded as a more useful specialization of Theorem (8.S). By making 
stronger assumptions, bounds are obtained that do not include || A* ||. 

(8.24) Theorem 

Use definitions of G„ i = 1,2, 3, from Eq. (8.7) to (9.9) and, in addi¬ 
tion, assume |]E||-|| A + 1| < 1 and Rank (A) <, Rank (A). Then Rank 
(A) — Rank (A) and 


(8.25) 

(8.26) 

(8.27) 

(8.28) 

where 


IIG.II^ 



II Gil! 


MMfCJi! 

i - lU 


wm 


IIG,||^||E|HM + |[* 


IIG||<£ 



(8.29) c = — 1618 if Ra °k (A) < min (m, n) 

(8.30) c = 2 Ui == 1.414 i/Rank (A) = min (m, n) < max (m, n) 

(8.31) c — 1 if Rank (A) = m = n 


Proof: The conclusion that Rank (A) = Rank (A) is obtained from 
Theorem (8.15). Using the bound for ||i? + || obtained in that theorem, one 
obtains Eq. (8.25), (8.26) and (8.27) from Eq. (8.10), (8.23) and (8.12), re¬ 
spectively. 

Thus the bound of Eq. (8.28) would follow immediately with c — 3. For 
practical purposes this would be a satisfactory result. The special result in 
Eq. (8.31) is also immediate since in this case ||(7 a || = ||<7,|| = 0. Of course 
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this result for a square nonsingular matrix is well known and can be proved 
more directly [e.g., see Wilkinson (1965a), pp. 189-190}. 

The special results of Eq. (8.29) and (8.30) are established as follows. Let 
x be a unit m-vector. Define 


Then 


= Qx 

*i = (/- Q)x 


x = x, + x a 

1-IWP-llJ.IP + IUalP 


and there exists a number f such that 


and 

Let 


cosp = |l*i II 
sinp = |l*all 


*;> 


HEMA+tt 

|£ 


1-W 


PPriT 


so that from the inequalities in Eq. (8.25) to (8.27) 


*^>I|G,|| / = 1,2,3 

Because of the rightmost factors A* and (/ — Q ) in Eq. (8.7) to (8.9), we 
have 

Gx — G,x t + G, Xj 4- G s x t 

= y t + yi + yi 

Because of the leftmost factors A* and (/ — P) in Eq. (8.7) to (8.9), the 
vector y t is orthogonal to y, and y t . Thus 


(8.32) 


llGx||* = ||y 1 +y 1 !i l + l|y,IP 

^« , l(ll*.ll + ||*alD J +||*«IP] 

= **[(cos f + sin p) 1 + cos* p] 

- «■(! + sin I, + i±“£) 

_ « a (3 + 2 sin 2y + cos 2p) 

^ « J (3 + 5 ,/a ) 

<-2- 
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Therefore 

II <7*11 ^ <L+ . ?* . *> = 1.618a 

Then 

|| G|| = max \\\Gx ||:||*||=1} 
which proves Eq. (8.29). 

For the conclusion in Eq. (8.30) we have either Rank (A) = n <m, in 
which case P — /„ so that G, = 0, or else Rank (/4) = m < n, in which case 
Q — I m so that G t = 0. Thus either y, or y 3 is zero in Eq. (8.32) leading to 

llGx||*^2a 1 

which establishes Eq. (8.30), completing the proof of Theorem (8.24). 

Equations (8.1) to (8.9) and the theorems of this chapter may be used to 
prove that with appropriate hypotheses on the rank of A the elements of A* 
are differentiable functions of the elements of A. 

Examples of some specific differentiation statements and formulas are 
given in Exercises (8.33) and (9.22M9.24). Note that the formulas given in 
these exercises generalize immediately to the case in which / is a ^-dimen¬ 
sional variable with components /„ • ••,/*. One simply replaces djdt in these 
formulas by djdt, for i = 1. k. 

Differentiation of the pseudoinverse has been used by Fletcher and Lill 
(1970) and by Perez and Scolnik (1972) in algorithms for constrained mini¬ 
mization problems. See Golub and Pereyra (1973) and Krogh (1974) for an 
application of differentiation of the pseudoinverse to nonlinear least squares 
problems in which some of the parameters occur linearly. 


EXERCISE 

(8.33) (Hanson and Lawson (1969); Pavel-Parvu and Korganoff (1969)] 
Let A be an m x n matrix with m ;> n whose elements are differen¬ 
tiable functions of a real variable t. Suppose that, for t — 0, A is of 
rank n. Show that there is a real neighborhood of zero in which A* 
is a differentiable function of t and the derivative of A* is given by 

T=- Atd -z A ' +A ' A 'f4)’v- AA ''> 



9 PERTURBATION BOUNDS FOR 

THE SOLUTION OF PROBLEM LS 


In this chapter the theorems of the preceding chapter will be applied to 
study the effect of perturbations of A and b upon the minimum length solu¬ 
tion x of Ax S b. We shall continue to use the definitions given in Eq. (8.1) 
to (8.4). Theorem (9.7) will be established without restriction on the relative 
sizes of m, n and k = Rank (A). Following the proof of Theorem (9.7) the 
bounds obtained will be restated for the three special cases of m — n — k, 
m> n — k and n> m — k. 

For convenience in stating results in terms of relative perturbations we 
define the relative perturbations 


(9.1) 


(9.2) 


a — 




_ I 


db 


1*1 


and the quantities 


(9.3) 

(9.4) 

(9.5) 

and 

(9.6) 


„ Pll .. 


M 

M*1 


p 

K 


- Ik 


IMII •11 *11 

U\\-\\A*\\ 




(r=»6 - Ax) 




K 


1 — Kd 1 — II E 




WT 
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The definitions in Eq. (9.1) to (9.6) of course apply only when the respective 
denominators are nonzero. The quantity ic of Eq. (9.S) is called the condition 
number of A. 

(9.7) Theorem 

Let x be the minimum length solution to the least squares problem Ax = 
b with residual vector r = b — Ax. Assume ||E||*||A + 1| < 1 and Rank 
(A) <; Rank (A), and let x + dx be the minimum length solution to the 
least squares problem 


(9.8) 


Then 


X{x + dx) = (A + £)(* + dx)cxb + db 
Rank (X) = Rank (4) 


(9.9) ||*f||£M + ||( 1 J||J!|!jfjlqr + 1 - Ilffi!u + II 




+ 


ll£||-ll*ll) 


and 


(9.10) 


dx 


jjL ^ + Ryfi + Kkpa + jco 

< *[(2 4- tcp)a + yfi] 


Proof: The conclusion in Eq. (9.8) follows from Theorem (8.15). The 
vectors x and x f dx satisfy 


and 

Thus 


x = A*b 


x + dx = X*(b + db) 

dx = X + (b + db) - A*b = (^ + - A*)b + X* db 
= Gb + X+ db 


Note that r — (I — Q)r = (/ — Q)b, which with Eq. (8.8) implies G 2 b — 
G t r. Then using Eq. (8.6) to (8.9) gives 

(9.11) dx = -X+Ex + G 2 r + (/ - P)E T A T *x + X * db 

Using the bound for |j X + 1| from Theorem (8.15) and the bound for || G 2 1| 
from Eq. (8.26), the result in Eq. (9.9) is obtained. 

Dividing the inequality in Eq. (9.9) by ||x||, using the definitions in Eq. 
(9.1) to (9.6) gives the inequality in Eq. (9.10). This completes the proof of 
Theorem (9.7). 
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Observe that when n — k = Rank (A), the matrix G, of Eq. (8.9) is zero, 
which leads to the fourth term in the right member of the inequalities in Eq. 
(9.9) and (9.10) being zero. Similarly when m = k = Rank ( A ), the matrix 
Gt of Eq. (8.8) and consequently the third term in the right member of the 
inequalities in Eq. (9.9) and (9.10) are zero. 

Furthermore if either n — kotm = k, then the rank of A clearly cannot 
exceed the rank of A. Thus the hypothesis Rank (A) <, Rank (A) that was 
used in Theorem (9.7) is automatically satisfied when either n = k or m — k. 
These observations provide proofs for the following three theorems. 

(9.12) Theorem 

Assume m > n = k = Rank (A) and ||E||»||A + || < 1. Then 


Rank (A) — Rank (A) 

(9.13, 11*11 S 


and 


(9.14) 


1 


j§^ <S *1(1 + K P )a + y0\ 


(9.1S) Theorem 

Assume m = n — k =~ Rank (A) ®u/||E||-|jA + 1| < I. Then 


(9.16) 


Rank (A) = Rank (A) 


and 


(9.17) 


j + Iff) <.*(* + P) 


(9.18) Theorem 

Assume n > m = k = Rank (A) om/||E||*||A 4 ’|| < 1. Then 


(9.19) 

(9.20) 


Rank (A) = Rank {A) 

\\dx \\<L \\A* || + Il£||-Il*ll) 


and 

+Yfi) + k* ^ *{2a + yfi) *(2a + 0) 
Alternatively, by an argument similar to that used to establish Eq. (8 JO), 
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it can be shown that 

(9.21) ^ *(2‘'»« + P) 

The following exercises illustrate some differentiation formulas that can 
be derived from the results of this chapter. See the remarks preceding Exer¬ 
cise (8.33) for an indication of generalizations and applications of these for¬ 
mulas. 


EXERCISES 

(9.22) Let A be an m x n matrix with m ;> n whose elements are differenti¬ 
able functions of a real variable t. Let x be a vector function of t 
defined by the condition that Ax^b for all t in a neighborhood U 
in which A is differentiable and Rank (A) == n. Show that for / e U, 
dx/dt exists and is the solution of the least squares problem 

4g) - -(£)*+ ^m T ' 

where r — b — Ax. 

(9.23) Further show that dx/dt is the solution of the square nonsingular 
system 

(9.24) If A = QFR where Q is n x m with orthonormal rows and R is n X 
n and nonsingular (a decomposition of A obtained by Householder 
transformations, for example), show that dx/dt satisfies 

"(f) - 



NUMERICAL COMPUTATIONS 
1 ft USING ELEMENTARY 
I W ORTHOGONAL TRANSFORMATIONS 


We now turn to describing computational algorithms for effecting an 
orthogonal decomposition of an m x n matrix A, as given in Theorem (3.19), 
and calculating a solution to Problem LS. 

Because of the variety of applications using Householder transformations 
or Givens transformations, we have found it convenient and unifying to 
regard each of these individual transformations as computational entities. 
We shall describe each of these in detail. These modules will then be used in 
describing other, more complicated, computational procedures. 

The computation of the Householder transformation can be broken into 
two parts: (1) the construction of the transformation and (2) its application 
to other vectors. 

The m x m Householder orthogonal transformation can be represented 
in the form 

(10.1) Q = /„ + b~'uu T 

where u is an m-vector satisfying ||n]| ^ 0 and b — —||u|| 2 /2. 

In Chapter 3 it was shown how for a given vector v a vector u could be 
determined so that Eq. (3.2) was satisfied. In practice, occasions arise in which 
one wishes to determine Q to satisfy 
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The computation of such a Q could be described as a row permutation follow¬ 
ed by an ordinary Householder transformation as described in Lemma (3.1) 
followed by a restoring row permutation. We find it more convenient, how¬ 
ever, to view Eq. (10.2) as defining the purpose of our basic computational 
module. The effect of the matrix Q in transforming v to y can be described by 
means of three nonnegative integer parameters, p, /, and m, as follows: 

1. If p > I, components 1 through p — 1 are to be unchanged. 

2. Component p is permitted to change. This is called the pivot element. 

3. If p < / — 1, components p + 1 through / — 1 are to be left un¬ 
changed. 

4. If / <; m, components l through m are to be zeroed. 

Note that we must have 

(10.3) 1 <,P<,m 
and 

(10.4) p < / 

The relation / > m is permitted in the Fortran implementation of this com¬ 
putational procedure (subroutine HI 2 in Appendix C) and is interpreted to 
mean that Q is to be an identity matrix. 

The computational steps necessary to produce an m x m orthogonal 
matrix Q that satisfies the conditions imposed by the integer parameters p. 
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/, and m can be stated as follows: 

(10.5) s — —o[v\ 4- vf) U1 where a — 

(10.6) u ( = 0 i = 1 . /» — 1 

(10.7) u f — v f — s 

(10.8) u, = 0 / = p 4 1,1 

(10.9) u, = v, i = t,... ,m 

( 10 . 10 ) b = su, 

(10.11) Q - /. 4 b-'uu T if h * 0 

= /„ if A — 0 


if v f ;> 0 
if », < 0 


The fact that the matrix g defined in Eq. (10.11) has the desired properties 
is established by the following three lemmas. 

(10.12) Lemma 

The m-vector u and the scalar b defined in Eq. (10.5) to (10.10) satisfy 

(10.13) b = 


Proof: This follows from the algebraic identities 

IMP-(•,-*)» + §•* 

= wj - 2v,s 4 s* 4 f\ vj 
= -2 s(t>, - s) 

— —2su, 

= -2b 


(10.14) Lemma 

The matrix Q of Eq. (10.11) is orthogonal. 

Proof: The verification that QfQ = I m follows directly using Eq. 
(10.11) and (10.13). 

(10.15) Lemma 

Let y — Qo. Then 


(10.16) 

(10.17) 


y, = v, 

y, = * 


i — 1». *«iP 1 
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(10.18) = / = />+1,...,/— l 

(10.19) y, — 0 i = /,..., m 

Proof: If v — 0, the lemma is obviously true. For v 0, the easily 
verified fact that u T v = —b while using Eq. (10.6) to (10.9) shows that 

y = Qv — v — u 

satisfies Eq. (10.16) to (10.19). 

In actual computation it is common to produce the nonzero components 
of the vectors u and y in the storage previously occupied by v. The exception 
to this is the pth entry in the storage array; a choice must be made as to 
whether u, or y, will occupy it. We shall let y p occupy it and store u p in 
an additional location; the quantity b of Eq. (10.10) can be computed from 
the identity indicated there whenever it is needed. 

After constructing the transformation one will generally wish to apply 

the transformation to a set of m- vectors, say, c,, j — 1.v. Thus it is 

desired to compute 

e s ~ Q c j J — !»•••»» 

Using the definition of Q given by Eq. (10.11), this computation is accom¬ 
plished as follows: 

(10.20) tj = b-'iScM 

(10.21) Cj^cj + tjul 1 , *"' V 

These computations will now be restated in an algorithmic form suitable 
for implementation as computer code. We shall define Algorithm Hl(p, /, m, 
v, h, C, v) for constructing and optionally applying a Householder transfor¬ 
mation and Algorithm H2(p, /, m, v, h, C, v) for optionally applying a pre¬ 
viously constructed Householder transformation. 

The input to Algorithm HI consists of the integers p, /, m, and v; the m- 
vector v, and, if v > 0, an array C containing the m-vectors c h j = 1,..., v. 

The storage array C may either be an m x v array containing the vectors 
Cj as column vectors or a v x m array containing the vectors c s as row vectors. 
These two possible storage modes will not be distinguished in describing 
Algorithms HI and H2. However, in references to Algorithm HI or H2 that 
occur elsewhere in this book we shall regard the column storage as normal 
and make special note of those cases in which the set of vectors c } upon 
which Algorithm HI or H2 is to operate are stored as row vectors in the 
storage array C. 

Algorithm HI computes the vector u, the number b, the vector y — Qv , 
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and, if v > 0, the vectors C } — Qc t ,j = 1,.... v. The output of Algorithm 
HI consists of the pth component of u stored in the location named h, the 
components / through m of u stored in these positions of the storage array 
named v, components 1 through / — 1 of y stored in these locations of the 
storage array named v, and, if v > 0, the vectors S„ j = 1,..., v, stored in 
the storage array named C. 

In Algorithm H2 the quantities p, /, and m have the same meaning as for 
Algorithm HI. The vector v and the quantity h must contain values computed 
by a previous execution of Algorithm HI. These quantities define a transfor¬ 
mation matrix Q. If v > 0, the array C must contain a set of m-vectors c } , 
j — 1,..., v, on input and Algorithm H2 will replace these with vectors 
Cj — Qcjtj = 1,..., v. 

(10.22) Algorithms Hl(p, /, m, v, h, C, v) [use Steps 1-11) and H2(p, /, m, 
v, h, C, v) [use Steps 5-11] 

Step Description 

( m \t/t 

+ 2 v t ) 

2 If Vj, > 0, set s := — s. 

3 Set h v, — s, v, s. 

4 Comment: The construction of the transformation is com¬ 
plete. At Step 5 the application of the transformation to 
the vectors c, begins. 

5 Set b := vji. 

6 If b = 0 or v = 0, go to Step 11. 

7 For j := 1,.... v, do Steps 8-10. 

8 Set s := (c fJ h + c t] v^jjb. 

9 Set c„ := c /} + sh. 

10 For i:—l,...,m, set c„ := c„ + sv,. 

11 Comment: 

(a) Algorithm HI or H2 is completed. 

(b) In Step 1 the computation of the square root of sum of 
squares can be made resistant to underflow by com¬ 
puting it based on the identity (wf + • • ■ 4- wi) l/2 = 
t[(wjty + • • • + (n' m /f)] l/2 , with t — max {| w,|, i — l, 
..., m}. 

The Fortran subroutine HI 2 given in Appendix C implements Algorithms 
HI and H2. 
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The other geaeral elementary orthogonal transformation we shall discuss 
is the Givens rotation. The formulas for construction of the Givens rotation 
were given in Eq. (3.S) to (3.9). The formulas for c and s given in Eq. (3.8) 
and (3.9) can be reformulated to avoid unnecessary underflow or overflow. 
This is accomplished by computing r = (x* + y 2 ) t/2 as 


(10.23) 


t = max(]x|,lyi) 


u = min([x[, |j>D 



On a machine that does not use normalized base 2 arithmetic, additional care 
should be taken to avoid loss of accuracy in computing c and s. For example, 
Cody (1971) has given the following reformulation of the expression in Eq. 
(10.23) for use with normalized base 16 arithmetic. 


ao.24, (s)T 

Algorithms Gl(e t , v 2 , c, s, r) and G2(c, s, z„ z 2 ) will be described for, 
respectively, constructing and applying a Givens rotation. The expression in 
Eq. (10.23), appropriate for base 2 arithmetic, will be used. The input to 
Algorithm G1 consists of the components v, and v 2 of the 2-vector v. The 
output consists of the scalars c and s, which define the matrix G of Eq. (3.5), 
together with the square root of sum of squares of v, and v 2 which is stored 
in the storage location named r. The storage location r can be identical with 
the storage location w, or » 2 , although it is most often convenient to identify 
it with the storage location for v,. The input to Algorithm G2 consists of the 
scalars c and s defining the matrix G of Eq. (3.5) and the components z, and 
z 2 of the 2-vector z. The output consists of the components d, and d 2 of the 
vector d—Gz stored in the locations called z, and z 2 . 

(10.25) Algorithm Gl(®„ t? 2 , c, r, r) 

Step Description 

1 If |v,| <; |w*l» go to Step 8. 

2 Set w := t> a /v,. 

3 Set? :== (1 + w 2 ) ,/2 . 

4 Set c := Ifq. 

5 If t>, < 0, set c := — c. 

6 Set s := wc. 

7 Set r := | «i |? and go to Step 16. 
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Step Description 

8 If 0, go to Step 10. 

9 Set c 1, s := 0, r := 0, and go to Step 16. 

10 Set w := vjv x . 

11 Setq := (1 + w 2 ) l/1 . 

12 Set s Ifq. 

13 If v x < 0, set s — s. 

14 Set c := ws. 

15 Setr:=|«*|q. 

16 Comment: The transformation has been constructed. 


(10.26) Algorithm G2(e, s,z it z 2 ) 

Step Description 

1 Set w z x c + z x s . 

2 Set z x — z x s + z x c. 

3 Set *, w. 

4 Comment: The transformation has been applied. 


The Fortran subroutines G1 and G2 given in Appendix C implement 
Algorithms G1 and G2, respectively. 

Many variations of algorithmic and programming details .are possible in 
implementing Householder or Givens transformations. Tradeoffs are possible 
involving execution time, accuracy, resistance to underflow or overflow, 
storage requirements, complexity of code, modularity of code, taking advan¬ 
tage of sparsity of nonzero elements, programming language, portability, etc. 
Two examples of such variations will be described. 

Our discussion of the Householder transformation has been based on the 
representation given in Eq. (10.1). The Householder matrix can also be 
expressed as 


(10.27) 


Q = I + gh r = I + 



[1, A„ 


AJ 


The representations in Eq. (10.1) and (10.27) are related by the substitutions 
g — b~ l u t u = s~ l u and A = «f'a. 

The form of Eq. (10.27) is mainly of interest for small m, in which case 
the need to store two vectors g and A instead of one vector u is of no con- 



60 ELEMENTARY ORTHOGONAL TRANSFORMATIONS 


CHAP. 10 


sequence and the saving of two multiplications each time a product € — Qc 
is computed is of some relative significance. This form is used with m — 3 in 
Martin, Peters, and Wilkinson [pp. 359-371 of Wilkinson and Reinsch (1971)] 
and with m = 2 and m — 3 in Moler and Stewart (1973). 

Specifically with m = 2 the computation of S — Qc using the expression 
in Eq. (10.1) [see Eq. (10.20) and (10.21)] requires five multiplications (or four 
multiplications and one division) and three additions, while the use of Eq. 

(10.27) requires only three multiplications and three additions. Thus Eq. 

(10.27) requires fewer operations than Eq. (10.1) and is, in fact, competitive 
with the Givens transformation, which, as implemented by Algorithm G2 

(10.26) , requires four multiplications and two additions to compute C = Gc. 
The actual comparative performance of computer code based on Eq. 

(10.27) versus code based on the conventional Givens transformation 
[Algorithms G1 (10.25) and G2 (10.26)] will very likely depend on details of 
the code. For example, we tested two such codes that used single precision 
arithmetic (27-bit fraction part) on the UNIVAC 1108 computer. These were 
used to zero the (2, l)-element of 100 different single precision 2x11 
matrices. For each of the 1100 2-vectors transformed, the relative error was 
computed as p = ||v' — ®"|!/il®'|l where »' is the transformed vector com¬ 
puted by one of the two single precision codes being tested and v" is the 
transformed vector computed by a double precision code. For the House¬ 
holder code based on Eq. (10.27), the root-mean-square value of p was 1.72 
x 2* 17 and the maximum value of p was 8.70 x 2 -17 . For the Givens code, 
the corresponding figures were 0.88 x 2 -17 and 3.17 x 2' 21 . 

Methods of reducing the operation count for the Givens transformation 
have been reported by Gentleman (1972a and 1972b). These methods require 
that the matrix A to be operated upon and the transformed matrix A —GA 
each be maintained in storage in a factored form. 

For ease of description of one of the methods given by Gentleman, con¬ 
sider the case of a 2 x n matrix A in which the (2, l)-element is to be zeroed 
by left multiplication by a Givens rotation. Thus 

(10.28) GA = A 

where &u — 0. Here the Givens matrix G would be constructed as 


(10.29) 


(10.30) 


r = (a] i + a|,) ,/J 



r 


r 

r 

«21 

Ou 

L r 

r _ 


P on 


if r > 0 
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In the technique to be described, instead of having the matrix A available 
in storage, one has matrices D 1k2 nod B ix „ such that 

(10.31) D = Diag {d u d 2 ) d,> 0 

and 

(10.32) A = D U1 B 

We wish to replace D and B in storage with new matrices D txl and S 2xn with 

(10.33) D = Diag [3 U J 2 } <?, > 0 

such that the matrix A of Eq. (10.28) is representable as 

(10.34) A = D l,1 $ or A = -D' /2 S 

We distinguish three cases: 

Case I. b ai = 0 

Case II. 0 < (d a b\,) <. (d,bU) 

Caseffl. O^^blXidibl) 

Cash I 

We may set 3 = D and B = B. 


Case II 
Define 

(10.35) 

(10.36) 

(10.37) 

(10.38) 

(10.39) 



Note that 6 U = fi n (l + 0 and B ai = 0. 
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Case III 
Define 


(10.40) 


(10.41) 


d t 

r+i 

(10.42) 


d x 

r+i 

hhi 

(10.43) 

//= 

“j6ji 

-1 

(10.44) 

3 — HB 

Note that £,, 

— 6»i(l + 0 and B u — 0. 


1 


hi 

b it . 


It is easily verified that the matrices D and 3 defined by this process 
satisfy Eq. (10.34). The saving of arithmetic in applying the transformation 
comes from the presence of the two unit elements in the matrix H. This per¬ 
mits the matrix multiplication B — HB to be done with 2 n additions and 2n 
multiplications in contrast to the requirement of 2n additions and 4n multi¬ 
plications required for the matrix multiplication A — GA. Furthermore the 
square root operation usually required to construct the Givens transforma¬ 
tion has been eliminated. 

In the case of triangularizing a general m X n matrix A by a sequence 
of Givens rotations, one could begin with D x — I mxm and B t — A and pro¬ 
duce a sequence of matrices [D k ] and {£*}, k = 1,2.by this procedure. 

In general the size of elements in D k will decrease as k increases but the rate 
is limited by the fact that £ <; (1 + f)“‘ 1. There is a corresponding but 

slower general increase in the size of the elements in B k since the euclidean 
norm of each column vector of the product matrix D\ n B k is invariant with 
respect to k. 

To reduce the likelihood of overflow of elements of B or underflow of 
elements of D, any code implementing this version of Givens rotations should 
contain some procedure to monitor the size of the numbers d„ rescaling d, 
and the ith row of B whenever d, becomes smaller than some tolerance. For 
example, one might set t = 2'**, p — t' 1 , and fi = x l/t . When a number d, 
is to be operated upon, it can first be compared with t. If d, < t, replace d, 
by pd t and replace b„, j = 1,.... n, by fib tJ ,j = 1,..., n. With this choice 
of t the multiplications by p and fi will be exact operations on many com¬ 
puters having base 2, base 8, or base 16 floating point arithmetic. 
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n THE OVERDETERMINED OR EXACTLY 
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In Theorem (3.11) we saw that for an m x n matrix A there existed an m 
x m orthogonal matrix Q such that QA — R is zero below the main diago¬ 
nal. In this chapter we shall describe the numerical computation of the 
matrices Q and R using Algorithms HI and H2 of Chapter 10. 

The matrix Q is a product of Householder transformations 

(11.1) £«£.'••£> 
where each Q } has the form 

(11.2) = L + b- t »«'%"«• j = 1.* 

Rather than saving the quantity b, that appears in Eq. (11.2), we shall 
recompute it when needed based on Eq. (10.10). 

Introducing subscripts we have 

(11.3) b, — Sju\ n 

Each Sj is the jth diagonal entry of the R matrix and will be stored as 
such. The quantities u < / > are stored in an auxiliary array of locations named 
bj,j 1,..., it. 

The computational algorithm will construct the decomposition of Theo¬ 
rem (3.11). This algorithm will be known as HFT(/«, n. A, h). The input to 
HFT consists of the integers m and n and the m x n matrix A. The output 
consists of the nonzero portion of the upper triangular matrix R stored in 
the upper triangular portion of the array named A, the scalars stored in 
the jth entry h t of the array named h, and the remaining nonzero portions 
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of the vectors u (J> stored as columns in the subdiagonal part of the yth column 
of the array named A. 

(11.4) Algorithm HFT(m, n, A, h) 

Step Description 

1 For j :^al, ... ,i», execute Algorithm HI (J,j + 1, m, a tJ , 
h}, a ltJ+u n — j) (see Algorithm 10.22). 

2 Comment: The (forward) triangularization is computed. 

In Step 1 of the algorithm above we are adopting the convention (con* 
sistent with Fortran usage) that the 7 th column of A can be referred to by 
the name of its first component, and the submatrix of A composed of columns 
7+1 through it can be referred to by the name of the first entry of column 

j + 1 * 

In Cases la and 2a of Fig. 1.1 the n x n upper triangular matrix R ,, (see 
Eq. 3.21) is nonsingular. Thus, in these two cases, the solution X of Problem 
LS can be obtained by computing 


(11.5) 

partitioning g as 

( 11 . 6 ) 

and solving 

(11.7) 


g=Qb 

, =M!" 

LStJJm ~ * 


for the solution vector X. 

Using Eq. (2.9) and (2.10), the residual vector r — b — AX and its norm 
can be computed as 


(11.8) r=e,.-. 

and 



(11.9) 


p = IMl = lls*ll 


Use of Eq. ( 11 . 8 ) and (11.9) obviates the need to save or regenerate the data 
matrix [A : 6 ] for the purpose of computing residuals. 

The following algorithm, HS1, will accomplish the computation of Eq. 
(11.5) to (11.7). The input to this algorithm will consist of the integers m and 
it, the arrays named A and h as they are output by the Algorithm HFT (11.4) 
and the array named b that holds the right-side m-vector b of Problem LS. 
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The output of the algorithm consists of the n-vector x replacing the first 
n entries of the array named b and, if m > n, the (m — »)-vector g 2 will 
replace entries n + 1 through m of the array named b. 

(11.10) Algorithm HSl(m, n, A, h, b) 

Step Description 

1 For j := execute Algorithm H2(j,j + 1, m, a u , 

hj, b, 1). 

2 Comment: In Steps 3 and 4 we shall compute the solution 
to the triangular system R tl x — g, of Eq. (11.7). 

3 Set b x := bja„. 

4 If » <. 1, go to Step 6. 

5 For i :=n — 1, n — 2,..., 1, seth, := (b, — Y a,.b.)la„. 

/*T+i 

6 Comment: The solution of Problem LS has been computed 
for Cases la and 2a. 

The case where b represents an m x l matrix can be easily handled by 

changing Step 1 to execute Algorithm H2 (J,j + l,m, a u , h Jt b, l),j =1. 

it, and by changing Steps 3 and 3 to deal with b as an (m x /)*array. 

To compute the residual norm p [see Eq. (11.9)], one could add to 
Algorithm HS1 

( m \ i/2 

2 tf) 

To compute the residual vector [see Eq. (11.8)], one could add to 
Algorithm HS1 

Step 8 For i 1.n, set b, := 0 

Step 9 For j := n, n— 1,.... 1, execute Algorithm H2 (J,j + 1, 

m, a u , hj, b, 1) 


Note that if Steps 8 and 9 are used, the solution vector x must first be 
moved to storage locations distinct from the array b if it is not to be over¬ 
written. Following Step 9 the residual vector r = b — Ax occupies the storage 
array called b. 

Algorithm HS1 is based on the assumption that the matrix A of Problem 
LS is of rank n. There is no test made in the algorithm to check for this. 

In practice it is important to know whether a change in the matrix A of 
the order of the data uncertainty could produce a matrix of rank less than n. 
One computational approach to this problem, involving the use of column 
interchanges as a first step, will be discussed in Chapter 14. Another 
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approach, which provides more detailed information about the matrix, 
involves the computation of the singular value decomposition. This will be 
treated in Chapter 18. 

Frequently, when a least squares problem is solved, there is also interest 
in computing the covariance matrix for the solution parameters. This topic 
will be treated in Chapter 12. 

A Fortran implementation of Algorithms HFT and HS1 of this chapter 
and the covariance matrix computation COV of Chapter 12 is given in 
PROG1 in Appendix C. This program constructs and solves a sequence of 
sample problems. 


EXERCISES 

(11.11) Derive a forward triangularization algorithm using Givens transfor¬ 
mations. Count the number of adds and multiplies separately. How 
is this count changed if the two-multiply, two-add transformation is 
used? 

(11.12) Determine the number of multiplications required to solve Problem 
LS using Algorithms HFT and HS1. Compare this count with that 
obtained for the Givens transformations of Exercise (11.11). 



COMPUTATION OF 
| J% THE COVARIANCE MATRIX 

I Jm OF THE SOLUTION PARAMETERS 


The symmetric positive definite matrix 

(12.1) C = (. A T A )-« Rank M) - n 

or a scalar multiple of it, say, o*C, has a statistical interpretation, under 
appropriate hypotheses, of being an estimate of the covariance matrix for 
the solution vector of Problem LS [e.g., see Plackett (I960)]. We shall refer 
to C as the unsealed covariance matrix. 

In this chapter some algorithms for computing C will be presented. We 
shall also identify some commonly occurring situations in which the explicit 
computation of C can be avoided by reconsidering the role of C in the later 
analysis. 

We shall not discuss the derivation or interpretation of the scalar factor a z 
but simply note that one expression commonly used to compute a* is 

( 12 . 2 ) a * = ~ - 11 * 

nt — n 


where X is the least squares solution of Ax S b and m and n are, respectively, 
the number of rows and columns of the matrix A. 

We shall discuss algorithms for computing C that make use of decom¬ 
positions of A. These decompositions occur in the course of three different 
methods for solving Problem LS that are treated in detail in this book. These 
three decompositions may be summarized as follows: 
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(12.3) 

(12.4) 





(12.5) U T AV=\ S 

|_o_ 


21,x. upper triangular (see Chapter 11) 


R hx . upper triangular (see Chapter 14) 


S mXK diagonal (see Chapter 18) 


The decompositions of Eq. (12.3) to (12.5) are, respectively, those ob¬ 
tained by Algorithms HFT (11.4) and HFTI (14.9) and the singular value 
decomposition of Chapter 18. The matrix Q of Eq. (12.3) is n x n and 
orthogonal. The matrices Q and P of Eq. (12.4) are both orthogonal, while 
P is a permutation matrix. The matrices U and V of Eq. (12.5) are, respec¬ 
tively, m x m and n x n, and they are orthogonal. 

It is easy to see that in the cases of Eq. (12.3) to (12.5) we have 

(12.6) A T A = RTR 

(12.7) A T A = PR t RP t 

(12.8) A T A = KS l F T 

so that if A is of rank n, inversion of these equations, respectively, yields 

(12.9) C = (A T A)- 1 = R-'(R~ l ) T 

(12.10) C = (A T AY l = PR-'(R-') T p r 

(12.11) C = C 4 r A )-* = VS~*V T 

We first consider the details of the computation required by Eq. (12.9) 

and (12.10). There are three basic steps. 

1. Invert the upper triangular matrix R or R onto itself in storage. 

2. Form the upper triangular part of the symmetric matrix R~ t (R~ l ) T or 
R~ l (R~ t ) T . This can replace R~ l or R~ l in storage. 

3. Repermute the rows and columns of the matrix R~'(R~') T . At each 
step the matrix is symmetric so only the upper triangular part of it need be 
saved. 


To derive formulas for the inversion of a triangular matrix R, denote 
the elements of R~ l by t tj . Then from the identity R~'R — I and the fact 
that both R and R ~ 1 are upper triangular, one obtains the equations 



if i = j 

if/*/ 


for \ i <j<,n. 
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Solving for t„ gives 



if f = i 
if />* 


In order to be able to replace r„ by t tJ in computer storage when t a is com¬ 
puted and to reduce the number of division operations, these formulas will 
be used in the following form: 


t„ = rr , 1 f=l,...,» 

^ tf]T IJy / = i "f 1,..., rtj l = 1, . • . , /> 1 

This set of formulas requires (n 2 f6) + 0(n 2 ) operations where an operation is 
a multiply plus an add or a divide plus an add. The postmultiplication of R ~ 1 
by its transpose indicated in Eq. (12.9) also requires (n*/6) + 0(n 2 ) opera¬ 
tions. Thus Algorithm COV requires (n J /3) + 0{n 2 ) operations to compute 
the elements of the upper triangle of (A r A)~‘ from the elements of 21. 

For Algorithm COV the matrix R or R of the right member of Eq. (12.6) 
or (12.7), respectively, must occupy the upper triangular part of an array 
named A on input. On output the upper triangular part of the matrix (AM) -1 
occupies the upper triangular part of the array named A. The array named p 
will hold the information (generated by Algorithm HFTI (14.9), for example] 
describing the effect of the matrix P of Eq. (12.10). 

(12.12) Algorithm COV(A, n, p) 

Step Description 


1 

For i :=* 

1,... 


set a„ := l/ar„. 

2 

If«= 1, 

goto 

Step 8. 

3 

For i := 

1,... 


— 1, do through Step 7 

4 

For/ := 

/+ I 

\ • • 

., n, do through Step 7. 

5 

Set s := 

0. 



6 

For/:= 

• 

1) • • • 

• 

- 1, set s :=* s + Otfiij. 

7 

Set a„ := 

- a jj S * 


8 

For i := 

1 9 • • ■ 


do through Step 12. 

9 

For/ := 

• 

Ij • • • 

,n. 

do through Step 12. 

10 

Set s := 

0. 



11 

For/ := 

• 

Jf • • • 


set s := s + a,fi n . 

12 

Set a,, := 

= s. 
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Step Description 

13 Remark: This completes the computation of the elements 
of the upper triangle of (A T Ay i for the case of Eq. (12.3) 
where no permutations were used in computing R. Alter¬ 
natively, in the case of Eq. (12.4) Steps 14-23 must be 
executed to accomplish the final premultiplication and 
postmultiplication by P and P T , respectively, as indicated 
in Eq. (12.10). In this case we assume an array of integers 
p„ i— 1,..., n, is available recording the fact that the 
/th permutation performed during the triangularization of 
A was an interchange of columns i and p t . 

14 For / := n, n — .1,..., 1, do through Step 22. 

15 If p, = i, go to Step 22. 

16 Set k := p t . Interchange the contents of a„ and a kk . If 
/ = 1, go to Step 18. 

17 For / := 1./ — 1, interchange the contents of a„ 

and a, k . 

18 If k — / = 1, go to Step 20. 

19 For / := / + 1. k — 1, interchange the contents of 

a a and a lk . 

20 If k — n, go to Step 22. 

21 For / := k + 1interchange the contents of a„ 
and a kt . 

22 Continue. 

23 Remark: The computation of the unsealed covariance 

matrix is completed. 

Examples of Fortran implementations of this algorithm with and without 
column permutations are, respectively, given in the Programs PROG1 and 
PROG2 of Appendix C. 

If the singular value decomposition is used, as in Eq. (12.5), the unsealed 
covariance matrix C satisfies Eq. (12.11). Thus the individual elements of C 
are given by 

(12.13) = ± 

*-i s kk 

If the matrix Khas been obtained explicitly in an (n x n)-array of storage, 
then VS~ l can replace V in storage. Next the upper triangular part of C 
can replace the upper triangle of VS~ l in storage. This requires the use of 
an auxiliary array of length n. 
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Rrnnarka on Soma Altarnatfvaa to Computing C 

We wish to identify three situations in which the unsealed covariance 
matrix C is commonly computed. 

(12.14) a The matrix C or C~ l may occur as an intermediate quantity in 

a statistical formula. 

(12.15) a The matrix C may be printed (or otherwise displayed) to be 

inspected and interpreted by the problem originator. 

(12.16) a Some subset of the elements of the matrix C may be used inter¬ 

nally as control parameters for some automated process. 

In each of these cases we shall indicate an alternative approach that may 
be either more informative or more economical. 

First consider Case (12.14). Commonly, formulas involving C or C~ l 
contain subexpressions of the form 

(12.17) E = BCB T 
or 

(12.18) F= D T C~ X D 

The most efficient, anu numerically stable, approach to computing such 
expressions is generally found by regarding C or C~ l in its factored form, 

(12.19) C-R-\Rr') T 
or 

(12.20) C ~ 1 = R T R 
respectively. 

Thus one would evaluate Eq. (12.17) by first solvingfor Tin the equation 

YR — B 

and then computing 

E — YY T 

Similarly in the case of Eq. (12.18) one would compute 

X = RD 

followed by 


F= X T X 
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The important general principle illustrated by these observations is the 
fact that a factorization of C or of C~ l is available as shown in Eq. (12.19) 
and (12.20), respectively, and can be used to develop computational proce¬ 
dures that are more economical and more stable than those involving the 
explicit computation of C or C' 1 . 

These remarks above can be easily modified to include the case where 
the matrices C or C _1 involve the permutation matrix P as in Eq. (12.7) and 
(12.10), respectively. 

In Case (12.15) the problem originator is frequently investigating correla¬ 
tions (or near dependencies) between various pairs of components in the 
solution vector x. In this context it is common to produce an auxiliary 
matrix, 

(12.21) E = D~ i CD~ l 

where D is a diagonal matrix whose ith diagonal element is given by d u 
= cl' 1 . Then e„ = 1, /= 1,...,», and \e tj \ < 1, i = 1,..., n’,j — 1. n. 

The occurrence of an element e u , i j, close to 1 or — 1, say, e„ = 0.95, 
for example, would be taken as an indication that the ith andyth components 
of the solution vector x are highly correlated. Algebraically this corresponds 
to the fact that the 2 x 2 principal submatrix 



is near-singular. 

A weakness of this type of analysis is the fact that only dependencies 
between pairs of variables are easily detected. For example, there may be a 
set of three variables that have a mutual near dependence, whereas no two 
of the variables are nearly dependent. Such a set of three variables could be 
associated with a 3 x 3 principal submatrix of E such as 


( 12 . 22 ) 


1 -0.49 -0.49' 

-0.49 1 -0.49 

-0.49 -0.49 1 


Here no off-diagonal element is close to 1 or — 1 and thus no principal 2x2 
submatrix is near-singular but the 3 x 3 submatrix is near-singular. This 
matrix becomes singular if the off-diagonal elements are changed from —0.49 
to —0.50. 

Dependencies involving three or more variables are very difficult to 
detect by visual inspection of the matrix C or E. Such dependencies are 
revealed, however, by the matrix V of the singular value decomposition A = 
USV* (or equivalently of the eigenvalue decomposition A T A = KS*F T ). 



CHAP. 12 


THE COVARIANCE MATRIX OF THE SOLUTION PARAMETERS 73 


Thus if Sj is a singular value of A that is small relative to the largest singu¬ 
lar value s u then the corresponding columns v, and u t of V and U, respec¬ 
tively, satisfy 

(12.23) AVj — SjUj 
or 

(12.24) \\Av,\\ = s, 

The fact that (sjst) is small may be taken as a criterion of near-singularity 
of A, while the vector v t identifies a particular linear combination of columns 
of A that are nearly dependent. The visual inspection of columns of V asso¬ 
ciated with small singular values has been found to be a very useful technique 
in the analysis of ill-conditioned least squares problems. 

In Case (12.16) the point to be observed is that one does not need to 
compute all the elements of C if only some subset of its elements are required. 
For example, once R ~ 1 is computed, then individual elements of C may be 
computed independently by formulas derivable directly from Eq. (12.9). 
Typical examples might involve a need for only certain diagonal elements of 
C or only a certain principal submatrix of C. 



COMPUTING THE SOLUTION 
m Q FOR THE UNDERDETERMINED 
I W FULL RANK PROBLEM 


Consider now the Problem LS, Ax = b, for the case m <n. Rank (A) 
= m (Case 3a, Fig. 1.1). 

A solution algorithm is described briefly in the following steps: 


(13.1) 

AQ = [R:0] 

(13.2) 

Ry, =b 

(13.3) 

y 2 arbitrary 

(13.4) 

X 

II 

s 

II 

to 

1 -1 


The algorithm will compute the orthogonal matrix Q of Eq. (13.1) so 
that the matrix R is lower triangular and nonsingular. The existence of such 
matrices was established by the transposed form of Theorem (3.11). 

The m-vector y, of Eq. (13.2) is uniquely determined. With any (n — m)- 
vector y t of Eq. (13.3), the vector x of Eq. (13.4) will satisfy Ax = b. The 
minimal length solution is attained by setting y 2 — 0 in accordance with 
Theorem (2.3). 

There are circumstances when this underdetermined problem Ax — b 
arises as a part of a larger optimization problem that leads to y 2 being non- 
zero. In particular, the problem of minimizing \\Ex — f\\ subject to the 
underdetermined equality constraints Ax = b is of this type. This problem, 
which we call LSE, will be treated in Chapters 20 to 22. 

The matrix Q of Eq. (13.1) is constructed as a product of the form 

(13.5) G=G. 
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where each Q t has the form 

(13.6) Qj -7. + b-jW'uW 7=1.m 

The quantity b, that appears in Eq. (13.6) is recomputed when needed 
based on Eq. (10.10). Introducing subscripts, we have 

(13.7) b J = 5 i u\\ n j = \,... ,m 

Each Sj is the y'th diagonal entry of the R matrix and will be stored as such. 
This algorithm will be known as HBT(m, n. A, g •). The input to Algorithm 
HBT consists of the integers m and n and the m x n matrix A of rank m. 
The output consists of the nonzero portion of the lower triangular matrix R 
stored in the lower triangular portion of the array named A. The u ( j n are 

stored in an auxiliary array named g Jt j = 1. m. The remaining nonzero 

portions of the vectors u w are stored in the upper diagonal part of the yth 
row of the array named A. 

(13.8) Algorithm HBT(m, n, A, g) 

Step Description 

1 For j:— 1,..., m, execute Algorithm Hl(/, j + 1.»,«m* 
gj, a y+ i,i, m — j ) (see Algorithm 10.22). 

2 Comment: The (backward) triangularization is computed. 

In Step 1, it should be noted that a Jt , denotes the start of a row vector in 
storage, while a J+ut denotes the start of m—j rows to be operated on by 
the transformations. 

The following algorithm will accomplish the computation of Eq. (13.2) 
to (13.4). The input of this algorithm will consist of the integers m and n, 
the arrays named A and g as they are output by Algorithm HBT (13.8). 
The solution will be returned in an array named x. The array b is replaced in 
storage by the vector y, of Eq. (13.2). 

(13.9) Algorithm HS2(m, n. A, g, 6, x) 

Step Description 

1 Set b, t=6|/ii||. 

(-i 

2 For i := 2,..., m, set b, := (b, — a,jbj)la u . 

3 Comment: At this point the vector y t must be determined. 
If the minimal length solution is desired, put y 2 : = 0. 

Set x := 
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Step Description 

5 For j 1,...»1, execute Algorithm H2(j,j + 1, 

ft , Oj » Si, X , 1). 

6 Comment: The array named x now contains a particular 
solution of the system Ax — b. If the vector y 2 of Step 4 is 
zero, the array x contains the solution of minimal length. 



COMPUTING THE SOLUTION FOR 

M PROBLEM LS WITH POSSIBLY 

DEFICIENT PSEUDORANK 


The algorithms of Chapters 11 to 13 for full rank problems are intended 
for use in cases in which the matrix A is assumed to be sufficiently well con¬ 
ditioned so that there is no possibility that either data uncertainties or virtual 
perturbations of A that arise due to round-off error during the computation 
could replace A by a rank-deficient matrix. Bounds for these latter perturba¬ 
tions will be treated in Chapters 15 to 17. 

There are other cases, however, when one would like to have the algo¬ 
rithm make a determination of how close the given matrix is to a rank-deficient 
matrix. If it is determined that the given matrix is so close to a rank-deficient 
matrix that changes in the data of the order of magnitude of the data un¬ 
certainty could convert the matrix to one of deficient rank, then the algorithm 
should take some appropriate action. 

The algorithm should at least inform the user when this near-rank-defi- 
cient situation is detected. In addition, one may wish to have a solution vec¬ 
tor computed by some procedure that avoids the arbitrary instability that 
can occur when a very ill-conditioned problem is treated as being of full rank. 

One technique for stabilizing such a problem is to replace A by a nearby 
rank-deficient matrix, say, A, and then compute the minimal length solution 
to the problem Ax s b. This replacement most commonly occurs implicitly 
as part of a computational algorithm. Algorithm HFTI, to be described in 
this chapter, is of this type. An example illustrating the use of Algorithm 
HFTI as well as some other stabilization methods is given in Chapter 26. In 
Chapter 25 other methods of stabilization are developed. 

We define the pseudorank k of a matrix A to be the rank of the matrix A 
that replaces A as a result of a specific computational algorithm. Note that 
pseudorank is not a unique property of the matrix A but also depends on 
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other factors, such as the details of the computational algorithm, the value 
of tolerance parameters used in the computation, and the effects of machine 
round-off errors. 

Algorithm HFTI applies in particular to the rank-deficient problems iden¬ 
tified as Cases 1 b, 2b, and 3b in Fig. 1.1. Strictly speaking, however, it is the 
pseudorank rather than the rank of A that will determine the course of the 
computation. 

Using Theorems (2.3) and (3.19), the mathematical relations on which 
Algorithm HFTI is based are the following: 

L 0 Rzi\)m—k 
k n — k 

®—tt-. 

[IF: 0] 

ny t = Cl 

y 2 arbitrary 

x = j = PK y 

II* - Ax\\ - ||cj - R 22 y 2 || (= ||Cj||ify 2 - 0) 

The algorithm will determine the orthogonal matrix Q and the permuta¬ 
tion matrix P so that R is upper triangular and R, , is nonsingular. 

The permutation matrix P arises implicitly as a result of the column in¬ 
terchange strategy used in the algorithm. The column interchange strategy is 
intimately related to the problem of determining the pseudorank k. An es¬ 
sential requirement is that the submatrix /?,, be nonsingular. Generally, 
unless other criteria are suggested by special requirements of the application, 
one would prefer to have reasonably well-conditioned and ||/? 22 || rea¬ 
sonably small. 

One example of an exceptional case would be the case in which A mx , is 
known to have zero as a distinct eigenvalue. Then one would probably wish 
to set k = n — 1 rather than have the algorithm determine k. 

Another exceptional case is the weighted approach to Problem LSE 
(Chapter 22) where it is reasonable to permit 7?,, to be very ill-conditioned. 

The column interchange strategy used in Algorithm HFTI is as follows. 
For the construction of the y'th Householder transformation, we consider 
columns j through n and select that one, say, column X, whose sum of squares 
of components in rows j through m is greatest. The contents of columns j 


(14.2) 

(14.3) 

(14.4) 

(14.5) 

(14.6) 

(14.7) 
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and A are then interchanged and the jth Householder transformation is con¬ 
structed to zero the elements stored in a IJt i — j + 1,..., m. 

Some saving of execution time can be achieved by updating the needed 
sums of squares. This is possible due to the orthogonality of the Householder 
transformation. The details of this appear in Steps 3-10 of Algorithm HFTI. 

As a result of this interchange strategy the diagonal elements of R will be 
nonincreasing in magnitude. In fact, they will satisfy the inequality in Eq. 
(6.16). In this connection, see Theorems (6.13) and (6.31). 

In Algorithm HFTI and in its implementation as the Fortran subroutine 
HFTI in Appendix C, the pseudorank k is determined as the largest index j 
such that \r ]i \> t, where x is a user-supplied nonnegative absolute tolerance 
parameter. 

The column-interchange strategy and the appropriate selection of r clearly 
depend on the initial scaling of the matrix. This topic is discussed in Chapter 
25. 

Choosing k to be less than min (m, n) amounts to replacing the given 


matrix 

< - c 

by the matrix 


(14.8) 

*- <o" of 

Note that 

11,?- XII = ||*22ll 

and 

II*- X||, = 11*2,11, 


Further, with the stated procedures for column interchange and pseudorank 
determination, one has 


<.(H — k) Ul T 

where 

It = min (m, n) 

The orthogonal matrix K of Eq. (14.3) is chosen so that IF is A; x k non¬ 
singular and upper triangular. The ^-vector y t is computed as the unique 
solution of Eq. (14.4). The (n — A)-vector y 2 can be given an arbitrary value 
but the value zero gives the minimal length solution. 

The final solution x is given by Eq. (14.6). The norm of the residual can 
be computed using the right side of Eq. (14.7). 
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The input to Algorithm HFTI will consist of the data A and b stored in 
the arrays called A and b, respectively, the integers m and ir, and the non* 
negative absolute tolerance parameter t. The orthogonal matrix Q of Eq. 
(14.1) is a product of ft Householder transformations Q,. The data that 
define these matrices occupy the lower triangular part of the array A on out¬ 
put plus ft additional stores in an array named k. 

The array h is also used to store the squares of the lengths of columns of 
certain submatrices generated during the computation. These numbers are 
used in the selection of columns to be interchanged. This information can be 
(and is) overwritten by the pivot scalars for the matrices Q,. 

The permutation matrix P is constructed as a product of transposition 
matrices, P — 0,Pi) • • • (ft,P„)- Here (/,/) denotes the permutation matrix 
obtained by interchanging columns / and j of /„. The integers p, will be re¬ 
corded in an array p. The orthogonal matrix K of Eq. (14.3) is a product of 
k Householder transformations K,. The data that define these matrices oc¬ 
cupy the rectangular portion of the array A consisting of the first k rows of 
the last a — k columns plus k additional stores in an array g. 

Figure 14.1 illustrates the output storage configuration for this decom¬ 
position when m = 6, n = 5, and k = Rank (A) — 3. If one wishes to 
retain complete information about the original matrix A, an extra array 
consisting of the diagonal terms of the matrix J? n of Eq. (14.1) must be 
recorded. These would be needed to compute the matrix Q of that same 
equation. Since Q is applied immediately to the vector b in Algorithm HFTI 
this additional storage is not needed. 




g: 



"'ll 

WI2 

Wl3 

*14 

*13 

*11 

W 22 

W23 

*24 

*23 

*13 

*23 

*33 

*34 

*33 

*14 

*24 

*34 

t 

t 

*15 

*23 

*33 

*43 

t 

«u 

*26 

*36 

*46 

*36 


Stores that initially held A. After 
processing they hold W and most of 
the information for constructing the 
Householder orthogonal decomposi¬ 
tion. 


*11 

*22 

*33 

*44 

*55 


*11 

*22 

*33 




Pi 

Pi 

Pi 

PA 

Pi 


Pivot scalars for the Householder 
transformations Qi 

1 Pivot scalars for the Householder 
transformations Ki. 


]} 


Interchange record. 


tlndicates elements of Rii that are ignored. 


Fig. 14.1 
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At the conclusion of the algorithm the solution vector x is stored in the 
storage array called x and the vector c is stored in the array called b. The 
algorithm is organized so that the names b and x could identify the same 
storage arrays. This avoids the need for an extra storage array for x. In this 
case the length of the b -array must be max (m, n). 

Steps 1-13 of Algorithm HFTI accomplish forward triangularization 
with column interchanges. This is essentially the algorithm given in Golub 
and Businger (1965). The additional steps constitute the extension for rank- 
deficient cases as given in Hanson and Lawson (1969). 

The coding details of Steps 3-10 are influenced by the code given in 
Bjdrck and Golub (1967). The test at Step 6 is essentially equivalent to “If 
h x > 10 J i jh.” The form used in Step 6 avoids the explicit use of the machine- 
dependent relative precision parameter if. 

(14.9) Algorithm HFTI(^, m, n, b, t, x, k, h, g, p ) 

Step Description 

1 Set p min (m, n ). 

2 For/:= 1. ft, do Steps 3-12. 

3 If j — 1, go to Step 7. 

4 For set ft,h, — a}.,.,. 

5 Determine A such that h A max {hj <, l <, n}. 

6 If (if + lO' 3 *,) > h, go to Step 9. 

HI 

7 For l :—j,... ,n, set h, := £ 3«• 

f-7 

8 Determine A such that h x :—max {h,:j <, l <^,n}. Set 
K := h x . 

9 Set pj A. If Pj — j, go to Step 11. 

10 Interchange columns j and A of A and set h x h,. 

11 Execute Algorithm HI {j,j + l, m, a tJ , h )t a tJ +„ n ~j). 

12 Execute Algorithm H2(J,j + 1, m, a lJt h } , b, 1). 

13 Comment: The pseudorank k must now be determined. Note 
that the diagonal elements of R (stored in a,, through a„„) 
are nonincreasing in magnitude. For example, the Fortran 
subroutine HFTI in Appendix C chooses k as the largest 
index j such that \aj,\> x. If all \ Ojj\< x, the pseudorank 
k is set to zero, the solution vector x is set to zero, and the 
algorithm is terminated. 

14 If k = n, go to Step 17. 
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Step Description 

15 Comment: Here k <n. Next, determine the orthogonal 
transformations K, whose product constitutes K of Eq. (14.3). 

16 For i := k, k — 1, ..., 1, execute Algorithm Hl(/, k+ 1, 
n, a tu g„ a, „ i — 1). (The parameters a n and a x , each iden¬ 
tify the first element of a row vector that is to be referenced.) 

17 Set x* := b k ja kk . If k <£ 1, go to Step 19. 

18 For i := k — 1, k — 2..1, x, := (b, — £ a u Xj)ja u 

[see Eq. (14.4)]. + ' 

19 If k = n, go to Step 22. 

20 Define the (n — k)-ve ctor y 2 of Eq. (14.5) and store its com¬ 
ponents in x„ i := k + 1,..., n. In particular, set y 2 0 
if the minimal length solution is desired. 

21 For i := 1 . k, execute Algorithm H2(i, k + 1, n, a n , 

g„ x, 1). [See Eq. (14.6). Here a tt identifies the first element 
of a row vector.] 

22 For j p, p — 1.1, do Step 23^ 

23 If p, ^ j, interchange the contents of x, and x fi [see Eq. 
(14.6)]. 

A Fortran subroutine HFTI, implementing Algorithm HFTI, is given in 
Appendix C. In the subroutine the algorithm has been generalized to handle 
multiple right-side vectors. The subroutine also handles the (unlikely) case 
of the pseudorank k being zero by setting the solution vector x to zero. 



■I JC ANALYSIS OF COMPUTING ERRORS 
19 FOR HOUSEHOLDER TRANSFORMATIONS 


Throughout the preceding chapters we have treated primarily the mathe¬ 
matical aspects of various least squares problems. In Chapter 9 we studied 
the effect on the solution of uncertainty in the problem data. In practice, 
computer arithmetic will introduce errors into the computation. The re¬ 
markable conclusion of floating point error analyses [e.g., see Wilkinson 
(1965a)] is that in many classes of linear algebraic problems the computed 
solution of a problem is the exact solution to a problem that differs from the 
given problem by a relatively small perturbation of the given data. 

Of great significance is the fact that these virtual perturbations of the data 
due to round-off are frequently smaller than the inherent uncertainty in the 
data so that, from a practical point of view, errors due to round-off can be 
ignored. 

The purpose of Chapters 15,16, and 17 is to present results of this type for 
the least squares algorithms given in Chapters 11,13, and 14. 

The set of real numbers that are exactly representable as normalized 
floating point numbers in a given computer will be called machine numbers. 

For our purposes the floating point arithmetic capability of a given com¬ 
puter can be characterized by three positive numbers L, U, and rj. The num¬ 
ber L is the smallest positive number such that both L and —L are machine 
numbers. The number U is the largest positive number such that both U and 
— U are machine numbers. 

In the analysis to follow we shall assume that all nonzero numbers x 
involved in a computation satisfy L < | x \ < U. In the practical design of 
reliable general-purpose programs implementing these algorithms the condi¬ 
tion |x| < U can be achieved by appropriate scaling. It may not be feasible 
to guarantee simultaneously that all nonzero numbers satisfy | x [ ;> L but 
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appropriate scaling can assure that numbers satisfying | jc | < L can safely 
be replaced by zero without affecting the accuracy of the results in the sense 
of vector norms. 

The number 17 characterizes the relative precision of the normalized float¬ 
ing point arithmetic. Following Wilkinson we use the notation 

-z = fl(x 4 - y) 

to indicate that z is the machine number produced by the computer when its 
addition operation is applied to the machine numbers x and y. If z = x + y, 
then even assuming L <, | z | < U, it is frequently true that z — z ^ 0. For 
convenience this difference will be called round-off error regardless of whether 
the particular computer uses some type of rounding or simply truncation. 

The number 7 is the smallest positive number such that whenever the true 
result z of one of the five operations, add, subtract, multiply, divide, or square 
root, is zero or satisfies L <,) z \ < U, there exist numbers e„ bounded in 
magnitude by 7 , satisfying the corresponding one of the following five equa¬ 
tions. 

(15.1) fl(x + y) = x(l + e 0 + jKI + e*) = 

(When jc and y have the same signs, e, — e, and e, = e 4 .) 

(15.2) fl(x -y) = x(l + e s ) - y( l + e 6 ) = 

(When x and y have opposite signs, e 5 = e t and e-, = f,.) 

fKxy) = xy( 1 + e 9 ) = T ^~ 

1 » Cjo 

"(7)^- (7)<' +' 

+ f „) = 

* T Cl« 

The general approach to analyzing algorithms of linear algebra using as¬ 
sumptions of this type is well illustrated in the literature [e.g., Wilkinson 
(1960) and (1965a); Forsythe and Moler (1967); Bjdrk (1967a and 1967b)]. 
Different techniques have been used to cope with the nuisance of keeping 
track of second- and higher-order terms in tf that inevitably arise. Since the 
main useful information to be obtained is the coefficient of 17 in the bounds, 
we propose to determine only that coefficient in our analyses. 

Due to grouping ? 2 terms into a single Olff*) term, the “bounds” obtained 
in Chapters 15,16, and 17 are not fully computable. For values of 7 corre- 


(15.3) 

(15.4) 

(15.5) 
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sponding to practical computers, however, the discrepancy from this source 
is negligible. Since maximum accumulation of errors is assumed throughout 
the analysis, the bounds obtained tend to be severe overestimates. The main 
practical purpose of these bounds is to give an indication of the dependence 
of the errors on the parameters m, n, and 7 and to establish the excellent 
numerical stability of the algorithms of Chapters 11,13, and 14. 

We first analyze the round-off error in Algorithm HI (10.22). This al¬ 
gorithm constructs a single m x m Householder transformation that will 
zero all but the first element of a given m-vector v. Including scaling to avoid 
loss of accuracy from underflow, the algorithm is mathematically defined by 


(15.6) 

ft = max, jo,| 

(15.7) 

' - £ ( 5 )' 

(15.8) 

• 

s = — (sgn v i)t m n 

(15.9) 


(15.10) 

u, = v, i = 2 ,... 

(15.11) 

6=2 su t 

(15.12) 

Q — /« + 6~'m T 


Using an overbar to denote computed machine numbers, the quantities 
actually computed using this algorithm may be written as 


(15.13) 

M — max, | Vf | 

(15.14) 

'-'{£ ($’] 

(15.15) 

S = -//[(sgn («,))(f) ,/, /il 

(15.16) 

«i =//(», - S) 

(15.17) 

11 

II 

IO 

• 

• 

9 

3 

(15.18) 

6 =//(»,) 

(15.19) 

Q = A/„ + 6-'wF ) 


The computed quantities defined in Eq. (15.14) to (15.19) are related to 
the true quantities of Eq. (15.7) to (15.12) as follows: 


(15.20) 

(15.21) 


f—t (I + t) |t|^ (m + 2 )(f + 0 ( 7 a ) 
j = j(l+<T) | g |^ (w + 6 )? + Q(7 a ) 

|*|<£<2L+ii2 + W ) 


(15.22) 


«, = «i,(l + K) 
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(15.23) u, = u, i = 2,... ,m 

(15.24) b = 6(1 + A) III <£ (« + 8)9 + Ofo*) 

(15.25) || C - CIIp <: 4k + 21 + 0(9*)^ (4m + 32)9 + 0(9*) 

The details of the derivation of these bounds are omitted since they are 
somewhat lengthy and are very similar to the derivation of bounds given on 
pp. 152-156 of Wilkinson (1965a). Note, however, that the bounds given 
here are different from those in Wilkinson (1965a) primarily because we are 
assuming that all arithmetic is done with precision 9 , whereas it is assumed 
there that some operations are done with a precision of 9 *. We defer discus¬ 
sion of such mixed precision arithmetic to Chapter 17. 

We next consider the application of a Householder transformation to an 
m x n matrix C using Algorithm H2 or the optional capability of Algorithm 
HI (10.22). Mathematically we wish to compute 

(15.26) B = QC 

In actual computation, however, we only have Q available in place of Q so 
the computed result will be 

(15.27) B =//(gC) 

The matrix B will satisfy 

(15.28) || B - 21||, - || [fl(QQ -QC] + (QC- QC )||, 

£ 11 fKGo - ecu, + ||0 ~ eimciu 
s|i£iu+ne-eiHicn, 

Using an analysis similar to that on pp. 157-160 of Wilkinson (1965a), it 
can be shown that 

(15.29) ||£||, £ (2m + S)||C||wf + 0(9*) 

Thus using Eq. (15.25), (15.28), and (15.29) gives 

(15.30) || B - B\\, ^ ( 6 m + 37)||C||rf + 0(9*) 

We must consider the error associated with the application of k successive 
Householder transformations beginning with anmxn matrix A. The true 
mathematical operation desired is 



(15.31) 

(15.32) 
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where the Householder matrix Q, is constructed to produce zeros in positions 
* + 1 through m of column i of Q,A,. This sequence of operations is basic to 
the various algorithms we are considering. 

The computed quantities will be 

(15.33) A, = A 

(15.34) A„ i = fliQJd / = 1,.... k 


where & is the computed approximation to the true matrix Q, that would 
have produced zeros in elements / + 1 through m of column i of the prod* 
uct matrix Q,A t . 

Define the error matrix 


(15.35) 


F, - Q<A t 


— Q<A t — A 


+f i 


i “ ly • • • f k 


From Eq. (15.30) we have the bound 

(15.36) UF'U,^ + Off, 2 ) 

where 

(15.37) = (6/+ 37)* 

It follows by direct substitution and the orthogonality of the matrices Q, 
that 

(15.38) |M*+i - G* • • • QiA\\, £ g M\, + (Kr, 1 ) 

^IMIlrg «*♦.-<+ <W) 

<£ (6m - 3k + 40)**jMI!i- + W) 


It is useful to define the error matrix 


— Q i • • • Qk*k*i — A 

Then 

(15.39) A k+l = Q„ • • ■ QM + H k ) 
and from Eq. (15.38) 

(15.40) || H k II, <.(6m-Zk + 40)*q|| A\\, + Otf) 
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Equation (15.39) and the bound in Eq. (15.40) can be interpreted as show¬ 
ing that the computed matrix A k+i is the exact result of an orthogonal trans¬ 
formation of a matrix A + H k where |j H k ||, is small relative to ||yf ||,. 

Note that the result expressed by Eq. (15.39) and (15.40) is independent 
of the fact that the matrices Q, were computed for the purpose of zeroing 
certain elements of A,. Equations (15.39) and (15.40) remain true for cases in 
which A (and thus A k+t ) is replaced by a matrix or vector not having any 
special relation to the matrices Q,. 

Furthermore if the computed matrices of Eq. (15.34) are used in the op¬ 
posite order, say, 

(15.41) ?-/*&&-••&£) 

where X is an arbitrary matrix, then the computed matrix Y satisfies 

(15.42) ? = Qi ••• Qk(X + JO 
where 

(15.43) || ATIU (6m - 3* + 40)*q|| AT||, + 0(t) 

The orthogonal matrices Q, in Eq. (15.42) are the same as those in Eq. (15.39). 
This latter observation will be needed in the proofs of Theorems (16.18), 
(16.36), (17.19), and (17.23). 

Finally we shall need bounds on the error in solving a triangular system 
of equations. Let R be a k x k triangular matrix and c be a ^-vector. Then 
the computed solution x of 

(15.44) Rx — c 
will satisfy 

(15.45) (R + S)x — c 

where 5 is a triangular matrix of the same configuration as R and satisfies 

(15.46) *11*11*+ W) 

For the derivation of this result, see Forsythe and Moler (1967), pp. 103-105. 
In the course of proving Eq. (15.46) in the place cited, it is shown that 

(15.47) |f H | <,24r tt \ + (Kt) 1 = 1.* 

so that for reasonable values of 9 we are assured that nonsingularity of R 
implies nonsingularity of Jl + S. 
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EXERCI8E 

(15.48) Let G denote a Givens matrix computed by Algorithm G1 (10.25) 
using exact arithmetic, and let G denote the corresponding matrix 
computed using arithmetic of finite precision tf. Let z be an arbitrary 
machine representable 2-vector. Derive a bound of the form 

|i/l(0i)-<b||£c|||j|| 
and determine a value for the constant c. 


1 M ANALYSIS OF COMPUTING ERRORS 
I O FOR THE PROBLEM LS 


In this chapter the error analyses of Chapter 1S will be used to obtain 
bounds for the computational errors of the algorithms of Chapters 11, 13, 
and 14. These algorithms provide solution procedures for the six cases of 
Problem LS defined in Fig. 1.1. 

It is assumed in this chapter that all arithmetic is done with the same 
precision 17 . This leads to error bounds involving ri 1 and mn. Smaller bounds, 
involving only the first power of n and independent of m, are obtainable if 
higher precision arithmetic is used at certain critical points in these compu¬ 
tations. The theorems of this chapter will be restated for the case of such 
mixed precision in Chapter 17. 

(16.1) Theorem (The full rank overdetermined least squares problem) 

Let A be an m X n matrix of pseudorank n (see Chapter 14 for defini¬ 
tion ofpseudorank) and b be an m -vector. IfX is the solution of Ax = b 
computed using Algorithms HFT (11.4) and HS1 (11.10) or HFTI 
(14.9) with computer arithmetic of relative precision t\, then X is the 
exact least squares solution of a problem 


(16.2) 

(A + E)x S 6 4- / 

with 



(16.3) 

ll£|i?^( 6 m- 

• 3n + 41)mj||i4||, + 0(ij*) 

and 



(16.4) 

II/II < ( 6 m — 

3n + 40)mf|| fr || 4- 0(q 2 ) 


90 
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Proof: The mathematical operations required may be written as 

06.5) &A 

(16.6) Rx = c 

Using Eq. (15.39) and (15.40) with A there replaced by the matrix [A: b] 
of Eq. (16.5), it follows that there exist an orthogonal matrix Q, a matrix G, 
and a vector / such that the computed quantities R, c, and d satisfy 

(16.7) CH + <?:6 + /] = [q Jj 
with 

(16.8) II<?||, ;£ (6m - 3n + Wpuft A\\, + Oft *) 
and 

(16.9) II/|| ^ (6m - 3* + 40)wf||6|| + 0(f) 

In place of Eq. (16.6) to be solved for x, the computer is presented with 
the problem 

Rx = c 

From Eq. (15.45) and (15.46) the computed solution x will satisfy 

(R + S)x = c 

with 

(16.10) l|5||,<£«9l|A||p + 0(f) = nii\A\\ r + 0(f) 

To relate this computed vector x to the original least squares problem, define 
the augmented matrix 

r* + s ci 

L o i\ 

Left multiplying this matrix by the matrix Q T where Q is the orthogonal 
matrix of Eq. (16.7) gives 
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Define 

Then x is the least squares solution of Eq. (16.2). Using ||£||^||G|| + J|S|| 
along with Eq. (16.8) to (16.10) establishes the bounds in Eq. (16.3) and 
(16.4). This completes the proof of Theorem (16.1). 

(16.11) Theorem (The square nonsingular problem) 

Let A be an n x n matrix of pseudorank n and b be an n -vector. 
Ifl is the solution of Ax = b computed using Algorithms HFT(11.4) 
and HS1 (11.10) or HFTI (14.9) with computer arithmetic of relative 
precision tf, then X is the exact solution of a problem 

(16.12) (A + E)x = b + f 
with 

(16.13) ||£||, <£ (3»> + 4\n)tf[\A\\f + Ofo*) 
and 

(16.14) ll/H <£ (3«* + 40»)*p|| + W) 

This theorem is simply a special case of Theorem (16.1) for the case n = m 
so the bounds in Eq. (16.13) and (16.14) follow from the bounds in Eq. (16.3) 
and (16.4). We have stated Theorem (16.11) as a separate theorem because 
there is often independent interest in the square nonsingular case. 

It is of interest to compare Theorem (16.11) with an analogous error 
analysis theorem for Gaussian elimination algorithms with pivoting [Forsythe 
and Moler (1967)]. 

In this case one has the inequality 

(16.15) ll*||~ ^ 1.01(n* + 3n^\A\Unp m 

Here the symbol || • !(„ denotes the max-row-sum norm defined for any 
m x n matrix B as 

||£||. = max 16//1:1 ^ t ^ m} 

The quantity p„ is a measure of the relative growth of elements in the course 
of Gaussian elimination (loc. cit., p. 102). 

If partial pivoting is used, one has the bound 


(16.16) 
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If complete pivoting is used, one has the bound 

(16.17) p.^l.8n«“« /4 

The usual implementations of Gaussian elimination employ the partial 
pivoting strategy. Thus the bound in Eq. (16.15) for the norm of the matrix E 
grows exponentially with n. 

With the use of the Householder algorithms such as Algorithm HFTI 
(14.9), the situation is much more satisfactory and assuring. In this case the 
inequality in Eq. (16.13) does not involve a growth term p x at all. Wilkinson 
(1965a) has also remarked on this but points out that roughly twice the num¬ 
ber of operations must be performed with the Householder algorithm as 
with Gaussian elimination with partial pivoting. Furthermore the quantity />„ 
can easily be computed in the course of executing a Gaussian elimination 
algorithm and Wilkinson has stated that in practice p n rarely exceeds 4 or 8 
or 16. 

Analogous remarks can be made about Theorem (17.15) where extended 
precision is used at certain points of the Householder algorithm. 

In Theorems (16.1) and (16.11) it was shown that the computed solution 
is the solution of a perturbed problem. Theorems (16.18) and (16.36), in 
contrast, show that the computed solution is close to the minimum length 
solution of a perturbed problem. This more complicated result is due to the 
fact that Theorems (16.18) and (16.36) concern problems whose solution 
would be nonunique without imposition of the minimum length solution 
condition. 

(16.18) Theorem (The minimum length solution of the full rank underdeter¬ 

mined problem) 

Let A be mm X n matrix ofpseudorank m and b be m m -vector. If 
X is the solution of Ax —b computed using Algorithms HBT (13.8) 
and HS2 (13.9) using computer arithmetic of relative precision if, 
then X is close to the exact solution of a perturbed problem in the 
following sense: There exists a matrix E md a vector Jt such that X 
is the minimum length solution of 

(16.19) (A 4- E)x — b 
with 

(16.20) II E\\, <, (6n - 3m + 41)mif|| A ||, + 0(f) 
and 

(16.21) || x - *|| ^ (6* - 3m + 40>w|||*|| + 0(f) 
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Proof: A brief mathematical statement of the algorithm of Chapter 

13 is 

(16.22) aQ = [R : 0] 

(16.23) Ry = b 

(16.24) * = 

In place of Eq. (16.22) the computed lower triangular matrix R will satisfy 

(16.25) (A + G)Q — [R : 0) 

where Q is orthogonal and, using the inequality in Eq. (15.40), 

(16.26) ||<7|tr <£ (6n -3 m + 40)«*|| A ||, + Ofo’) 

Furthermore from Eq. (15.45) and (15.46) it follows that in place of Eq. 
(16.23) the computed vector y will satisfy 


(16.27) 

(R + S)y — b 

with 


(16.28) 

IISII <£ + W) - mtf[\A\\ r + Ofo») 


In place of Eq. (16.24) the computed vector x will be defined by 

(16.29) * = C {[o] + A j 

where from Eq. (15.41) to (15.43) one obtains 

(16.30) \\h\\<.(6n-3m + 40)mrfi\y || + W) 

The orthogonal matrix Q in Eq. (16.29) is the same as the matrix Q in Eq. 
(16.25) as noted in the remarks associated with Eq. (15.41) to (15.43). 
Rewriting Eq. (16.27) as 

(16.31) [R + S: 0)[ J ] = b 
then 


(16.32) 


m 

[R + S:0Jg»-e y Q =6 
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and using Eq. (16.25) and (16.29) there follows 

(16.33) {A + G + (S:0]G r }(jf -Qh) = b 

Define 

(16.34) E=G + [S:0}Q r 

and 

(16.35) * = *-GA = e[o] 

From the inequality in Eq. (15.47) we may assume that R + S of Eq. 
(16.32) is nonsingular and thus that the row space of A + E = (it + S: 0]Q T 
is the same as the space spanned by the first m columns of Q. 

From Eq. (16.33) to (16.35), X satisfies Eq. (16.19), while again using Eq. 

(16.35) , X lies in the space spanned by the first m columns of Q. Thus X is 
the unique minimum length solution of Eq. (16.19). 

The matrix E and the vector (x — X) satisfy 

l|£||p £IIGIL- + ||S||, <£( 6 « - 3m + 4l)n4\A\\, + W) 

and 

II* - *11 = Pll^ (6* - 3« + 40)mi>||.>MI + OOl 1 ) 

— (6 n — 3 m + 40)m^||£|| + (Xjl 1 ) 

which establish the inequalities in Eq. (16.20) and (16.21), completing the 
proof of the Theorem (16.18). 

(16.36) Theorem {The rank-deficient Problem LS) 

Let A. be an m x n matrix and b bean m -vector. Let X be the solution 
of Ax s b computed using Algorithm HFTI ( 14.9) with computer 
arithmetic of relative precision q. Let k be the pseudorank determined 
by the algorithm and let R 21 be the computed matrix corresponding to 
R** of Eq. (14.1). Then X is close to the exact solution to a perturbed 
problem in the following sense: There exists a matrix E and vectors f 
and X such that X is the minimal length least squares solution of 

(16.37) (A + E)x^b+f 
with 

(16.38) 11*11 <£ ||* M IU + (6m + 6n - 6k - 3p + 84)^| \A ||, 

4- (Hi} 1 ) p = min (m, n) 
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(16.39) || /1| £ ( 6 « - 3* + mm 6 || + W) 

(16.40) II x — X || £ (6n - 6k + 43)*lf|| je|| + W) 

Proof: Refer to Eq. (14.1) to (14.6) for the mathematical description 
of the algorithm. 

Recall that the matrix Q of Eq. (14.1) is the product of n Householder 
transformations, Q = Q„ • • • Q\. Note that the matrices R x , and R Xi of Eq. 
(14.1) and the vector c x of Eq. (14.2) are fully determined by just the first k 
of these transformations. Let 

(16.41) Q = Q k • • • Q x 
Then 

<.«.«, *•>-[*■ *;] 

and 

(16.43) Qb - 
where 

(16.44) IISU,-Il*ssllr 
and 

(16.45) ||rf|| = ||ca|| 

In place of Eq. (16.42) the computed matrices R xx , R x2 , and S will satisfy 

(16.46) Q(A + G)/> = ‘ ^ 2 J 

where, using the results of Chapter 15, Q k is an exactly orthogonal matrix and 

(16.47) IIGIU ^ (|j «U*,-/)MIU + W) 
and 

\\S\\,<. II*.*II, + ( | _g i #«+i-,)l! J ^llr + W) 

Here a, is defined by Eq. (15.37), n — min (m, n) as in Chapter 14, and R 21 
denotes the computed matrix corresponding to R it of Eq. (14.1). 
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From Eq. (16.46) we may also write 

(16.48) Q(A + H)P = * 

with 

(16.49) || H |U < ||% + IIClip 11*2*11, + (£ «»*,-,)|MIIp + W) 

£ II *2i Up + (6m — 3/1 H- 40)^M||, + 0(*») 

In place of Eq. (14.3) the computed W satisfies 

(16.50) ft *, t : R ti ) + M}t =* [&: 0] 
where R is an orthogonal matrix. 

The bound for || A/||, is somewhat different than is given by Eq. (15.40) 
since each of the * Householder transformations whose product constitutes 
K has only n — k + 1 columns that differ from columns of the identity 
matrix. Therefore, using a, as defined in Eq. (15.37), 

(16.51) |M#||, ^ a.-**,**111*1,: *, J||, + Ofo 1 ) 

= (6« - 6* + 35)**|MIIp + Oin 2 ) 

In place of Eq. (16.43) the computed vectors c, and d satisfy 

(16.52) Q(b + /) = 

where Q is the same matrix as in Eq. (16.46) and 

(16.53) ||/1| ^ (6m - 3* + 40)**||6|| + Ofo 1 ) 

which will establish Eq. (16.39) of Theorem (16.36). 

Referring to Eq. (14.4) the computed vector/, will satisfy 

(16.54) (W + Z)/, = c, 
with (1 V + Z) nonsingular and 

(16.55) l|Z||,^**||lF||, + 0(**) 

^**IMIIp +W) 

In Eq. (14.5) we shall consider only the case y 2 — 0. In place of Eq. (14.6) 
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the computed x satisfies 


(16.56) 



+ h 


with 


(16.57) || *|| («» - 6ft + 43)**Uy, || + W) 


The error coefficient in Eq. (16.57) is determined in the same way as that 
in Eq. (16.51). 

Define 


(16.58) 


* - *[o'_ 


Then from Eq. (16.56) to (16.58) 


(16.59) II x - X || <£ (6/« - 6k + 43)**||jc || + 0(* a ) 

which establishes Eq. (16.40) of the theorem. 

It follows from Eq. (16.54) and (16.58), by arguments similar to those 
used in proving Theorem (16.18), that J i is the minimal length solution of 

(16.60) [lV + Z : 0]#7»x = c, 


From Eq. (16.50) 

(16.61) [>F+Z:0]tf r = + X 


where 

(16.62) X = M + [Z: O]#’ 


and from Eq. (16.51) and (16.55) 

(16.63) lUriu^HJfiu + iiziL- 

^ (6„ - 6k + 44)**|M |U + W) 


Substituting Eq. (16.61) into Eq. (16.60) and augmenting the system with 
additional rows shows that X is the minimal length least squares solution of 



(16.64) 
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Left-multiplying Eq. (16.64) by & T and using Eq. (16.46) shows that X 
is the unique minimum length solution of the least squares problem, Eq. 
(16.37), with 

(16.65) S=/r+g r [£]/> r 
Thus 

(16.66) ll*iu<;il/r||p +TO 

or using Eq. (16.49) and (16.63), and using k <, ft to simplify the final expres¬ 
sion, 

(16.67) ||f II, ^ 114,11, + (6m + 6n-6k-ifi + MtaMllr + W) 
With Eq. (16.53) and (16.59) this completes the proof of Theorem (16.36). 




ANALYSIS OF COMPUTING ERRORS 
FOR THE PROBLEM LS USING 
MIXED PRECISION ARITHMETIC 


In our analysis of computing errors we have assumed that all arithmetic 
was performed with the same precision characterized by the parameter i\. 
If a particular computer has the capability of efficiently executing arithmetic 
in higher precision (say, with precision parameter at < if) and has efficient 
machine instructions for converting numbers from one precision to the other, 
then it is possible to use this extended precision at selected points in the com¬ 
putation to improve the computational accuracy with very small extra cost 
in execution time and essentially no extra storage requirements. 

Generally, unless one decides to convert entirely from 17 -precision to 
co-precision, one will limit the use of co-precision arithmetic to those parts of 
the algorithm that will require only a small fixed number of co-precision 
storage locations, the number of such locations being independent of the 
problem dimensions m and n. 

Wilkinson has published extensive analyses [e.g., Wilkinson (1965a)] of 
computing errors for specific mixed precision versions of algorithms in 
linear algebra assuming o> — if 2 . We shall use similar methods of analysis to 
derive error bounds for specific mixed precision versions of the algorithms 
analyzed in the previous chapter. 

For the computation of a Householder transformation we propose to 
modify Eq. (15.13) to (15.19) as follows. Equation (15.14) will be computed 
using co-precision arithmetic throughout, with a final conversion to represent 
t as an 17 -precision number. Then the error expressions of Eq. (15.20) to 
(15.25) become 

(17.1) f=t(l+T) + 
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(17.2) I = -Kl + a) \e\^% + OQft 

(17.3) «, = 11,(1 + k) |*| <^ + <Kn *) 

(17.4) b = b(l+fi) |0|<£ 7//+ <*//*) 

(17.5) II5-fill, <£28// + W) 

In place of Eq. (15.27) the matrix 

(17.6) B = fl[C + &(&€$-'] 

will be computed using co-precision arithmetic with a final conversion of the 
elements of B to //-precision form. Then in place of the error bounds in Eq. 
(15.29) and (15.30) we have, respectively, 

(17.7) ll*llr£l|C||rf + W) 
and, using Eq. (17.5), 

(17.8) ||*-*||, <£29 ||C||rf+ W) 

Then the bound in Eq. (15.40) on the error associated with multiplication 
by * Householder transformations becomes 

(17.9) II tf* II, <£ 29* M||rf+ <*//*) 

Finally, co-precision arithmetic could be used in solving the triangular 
system of Eq. (15.44) in which case the bound in Eq. (15.46) would become 

(17.10) ll$lt^ll*lt* + W) 

The error bound in Eq. (15.46) is already small relative to the error bounds 
from other sources in the complete solution of Problem LS and thus the addi¬ 
tional reduction of the overall error bounds due to the use of co-precision 
arithmetic in solving Eq. (15.44) is very slight. For this reason we shall assume 
that //-precision rather than co-precision arithmetic is used in solving Eq. 
(15.44). 

With this mixed precision version of the algorithms, Theorems (16.1), 

(16.11) , (16.18) and (16.36) are replaced by the following four theorems. 
Proofs are omitted due to the close analogy with those of Chapter 16. 

(17.11) Theorem (The full rank overdetermined least squares problem ) 

Let A be an m x n matrix of pseudorank n and b be an m -vector. 
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IfX is the solution of Ax s b computed using Algorithms HIT (11.4) 
and HS1 (11.10) or HFTI (14.9), with mixed precision arithmetic of 
precisions tf and to <, if as described above , then X is the exact least 
squares solution of a problem 

(17.12) (A + E)x^b + f 
where 

(17.13) ||£|U^30n|M||,»f + O(7 l ) 

and 

(17.14) || /1| <£29« || b || 7 + <W 

(17.15) Theorem (The square nonsingular problem) 

Let A be an n x n matrix ofpseudorank n and b bean n -vector. Iff. 
is the solution of Ax — b computed using Algorithms HFT (11.4) and 
HS1 (11.10) or HFTI (14.9) with mixed precision arithmetic of preci¬ 
sions tf and as described above, then X is the exact solution 

of a problem 

(17.16) (A + E)x = b +f 
with 

(17.17) \\E\\ r ^m\\AU + 0(t,*) 
and 

(17.18) II / II, ^ 29« ||61| 17 + W) 

(17.19) Theorem (The minimum length solution of the full rank underdeter¬ 

mined problem) 

Let A be an m x n matrix of pseudorank m and b be an m -vector. 
If His the solution of Ax = b computed using the Algorithms HBT 
(13.8) and HS2 (13.9) with mixed precision arithmetic of precisions tf 
and as described above, then x is close to the exact solution of 
a perturbed problem in the following sense: There exists a matrix E 
and a vector X such that X is the minimum length solution of 

(17.20) (A + E)x = b 
with 

(17.21) ||£||,£3aw||i4|fcf + W) 

and 

(17.22) \\x - X\\^29m\\x\\n + W) 
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(17.23) Theorem (The rank-deficient Problem LS) 

Let A be an m x n matrix and bbean ta-vector. Let X be the solution 
of Ax = b computed using Algorithm HFTI (14.9) with mixed preci¬ 
sion arithmetic of precisions * and <o<,q* as described above. Let k 
be the pseudorank determined by the algorithm and let R 22 be the 
computed matrix corresponding to R 22 of Eq. (14.1). Then X is close 
to the exact solution to a perturbed problem in the following sense: 
There exists a matrix E and vectors f and X such that X is the minimal 
length least squares solution of 


(17.24) 

(A + E)x^b+f 

with 


(17.25) 

II E\\r £ II Rn\\r + 59* \\A ||rf + 0(**) 

(17.26) 

H/ll £ 29*||6||*+ 

(17.27) 

||i - *|| <S 29* j|x|| * + <**») 


A different type of round-off error analysis of the Householder trans¬ 
formation algorithm as applied to the problem A mxm x ^b,m^n. Rank (A) 
~ n, has been carried out by Powell and Reid (1968a and 1968b). This analy¬ 
sis was motivated by consideration of the disparately row-weighted least 
squares problem which arises in the approach to the equality-constrained 
least squares problem described in Chapter 22. 

In Chapter 22 we shall wish to solve a least squares problem in which 
some rows of the data matrix [A: b] have the property that all elements of the 
row are very small in magnitude relative to the elements of some other rows 
but the relative accuracy of these small elements is nevertheless essential to 
the problem. By this it is meant that perturbations of these elements that are 
significant relative to the size of the elements will cause significant changes in 
the solution vector. 

Clearly, Theorem (17.11) does not assure that the Householder algorithm 
will produce a satisfactory solution to such a problem since Eq. (17.13) and 
(17.14) allow perturbations to all elements of A and b which, although “small” 
relative to the largest elements of A and b, respectively, may be significantly 
large relative to the small elements. 

Indeed Powell and Reid found experimentally that the performance of 
the Householder algorithm [such as Algorithm HFTI (14.9)] on disparately 
row-weighted problems depended critically on the ordering of the rows of the 
matrix [A: b ]. They observed that if a row of small elements was used as 
the pivotal row while some of the following rows contained large elements in 
the pivotal column, then in Eq. (10.7) the quantity v„ would be small relative 
to s. In the extreme case of J v„\j\ s | < * the quantity v, makes no contribution 
in Eq. (10.7). Furthermore in such cases it may also be true that the pth 
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component of Cj will be much smaller in magnitude than the /rth component 
of tjU in Eq. (10.21) so that the pth component of c, makes no contribution in 
Eq. (10.21). Thus if the elements of the pivotal row are very small relative to 
the elements of some of the following rows, the effect can be essentially the 
same as replacing the data of the pivotal row with zeros. If the solution is 
sensitive to this loss of data, then this situation must be avoided if possible. 

Powell and Reid observed satisfactory results in such problems after 
adding a row-interchange strategy, which we shall describe preceding Theo¬ 
rem (17.37). 

We shall state three theorems given in Powell and Reid (1968a) that pro¬ 
vide insight into the performance of Householder transformations in dis- 
parately row-weighted problems. 

For notational convenience, assume the columns of A are ordered a priori 
so that no column interchanges occur during the execution of Algorithm 
HFTI. Let = [A : b] and let &?*♦»»: £“♦»] denote the computed 

matrix resulting from application of the fth Householder transformation to 
[A w : b U) ] for / = l,...,n. 

Define 

(17.28) y, = max | oft* | i = 1. tn 

where a }) 1 is the (/,y)-element of A <k \ 

Let Q iki denote the exact Householder orthogonal matrix determined by 
the condition that elements in positions (/, k),i — k + 1 ,..., m, in the prod¬ 
uct matrix Q {k) A ik) must be zero. 

(17.29) Theorem [Powell and Reid (1968a) ) 

Let A be an m x n matrix of pseudorank n. Let A be reduced to 
an upper triangular m x n matrix A (a+,) by Algorithm HFTI (14.9) 
using mixed precision arithmetic of precisions if and <o <, tf 1 . Then 

(17.30) A ( '+'> = Q<*>Q<”-'> - • • Q"'(A + E) 


where the elements e u of E satisfy 

(17.31) \e,j\<. (20n* - 17* + 44)jw + Ofo*) 

For the statement of Theorem (17.32) we need the following additional 
definitions to account for the fact that the right-side vector b cannot partici¬ 
pate in the column interchanges: 


ft, = max | B} k) | /=!,...,»! 



k = 1,... 


n 
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WCT*}*" * = 

P = max (max, max* ^ 

Note that because of our assumptions on the a priori ordering of the columns 
of A, the quantity a k satisfies 

<r* = max (Jj k = 1. n 

(17.32) Theorem [Powell and Reid (1968a)] 

Let A be an m X n matrix of pseudorank n and let b bean m -vector. 
Let X be the solution o/ Ax = b computed using Algorithm HFTI 
(14.9) with mixed precision arithmetic of precisions if and m <L f. 
Then X is the exact least scpiares solution of a problem 

(17.33) (A + E)x?zb+f 
where the elements e u of E satisfy 

(17.34) \e„\<, (20n* - 1 Sn + 44)m + 0(f) 
and the elements f, off satisfy 

(17.35) |/ l |<S(20«’ + 2Qn + 44 )pypi + 0(f) 

Using the general fact that \e„\ <; ||£||„ one can obtain the inequality 

(17.36) \e,j\£30n\\A\\rt + O(f) 

from Eq. (17.13). Comparing inequalities in Eq. (17.34) and (17.36) we note 
that inequality in Eq. (17.34) will be most useful in those cases in which y, is 
substantially smaller than Mil,. Analogous remarks hold regarding Eq. 
(17.14) and (17.35). 

Recall that y, denotes the magnitude of the largest element occurring in 
row / of any of the matrices A " 1 ,..., A ,m+, \ Thus for y, to be small (relative 
to II A ||,) requires that the elements of row i of the original matrix A iU — A 
must be small and that elements in row i of the successive transforming 
matrices A {k) must not grow substantially in magnitude. 

Powell and Reid have given an example showing that if no restriction is 
placed on the ordering of the rows of A, the growth of initially small elements 
in some rows can be very large. They recommend the following row-inter¬ 
change strategy: Suppose that a pivot column has been selected and moved 
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to the kth column of the storage array in preparation for the kth Householder 
transformation. Determine a row index / such that 

|a,*| = max {U,*|: * ^ ^ m) 

Interchange rows l and k. The &th Householder transformation is then 
constructed and applied as usual. This process is executed for k — 1.n. 

(17.37) Theorem [Powell and Reid (1968a)] 

With y, defined by Eq. (17.28) and using the column- and row-inter¬ 
change strategy just described, the quantities y t satisfy 

*<£(1 +2 ,/a )*- 1 m ,/I max,|a}}M 


for 1 ^ i m. 

Powell and Reid report that using this combined column- and row-inter¬ 
change strategy they have obtained satisfactory solutions to certain dispa- 
rately row-weighted problems for which results were unsatisfactory when 
rows of small elements were permitted to be used as pivotal rows. 



COMPUTATION OF THE SINGULAR 
1 fi VALUE DECOMPOSITION AND THE 
I O SOLUTION OF PROBLEM LS 


Section 1. Introduction 

Section 2. The Symmetric Matrix QR Algorithm 
Section 3. Computing the Singular Value Decomposition 
Section 4. Solving Problem LS Using SVD 
Section 5. Organization of a Computer Program for SVD 


Section 1. INTRODUCTION 

A review of the singular value decomposition of a matrix was given in 
Golub and Kahan (1965). Included was a bibliography dealing with applica¬ 
tions and algorithms, and some work toward a new algorithm. This work 
was essentially completed in Businger and Golub (1967). An improved ver¬ 
sion of the algorithm is given by Golub and Reinsch in Wilkinson and 
Reinsch (1971). This algorithm is A. special adaptation of the QR algorithm 
[due to Francis (1961)] for computing the eigenvalues and eigenvectors of a 
symmetric matrix. 

In this chapter we present the symmetric matrix QR algorithm and the 
Golub and Reinsch algorithm (slightly enhanced) for computing the singular 
value decomposition. 

A = U pS'"'"! yr 

n »ix« v mxm q I v ax* 

Included are additional details relating to the use of the singular value decom¬ 
position in the analysis and solution of Problem LS. 

For notational convenience and because it is the case in which one is 
most likely to apply singular value analysis, we shall assume that m > n 
throughout this chapter. The case of m < n can be converted to this case by 
adjoining n ~ m rows of zeros to the matrix A or, in the case of Problem 
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LS, to the augmented data matrix [A : b]. This adjoining of zero rows can be 
handled implicitly in a computer program so that it does not actually require 
the storing of these zero rows or arithmetic operations involving these zero 
elements. Such an implementation of singular value decomposition with 
analysis and solution of Problem LS is provided by the set of Fortran sub¬ 
routines SVA, SVDRS, and QRBD in Appendix C. 


Section 2. THE SYMMETRIC MATRIX OR ALGORITHM 

The symmetric matrix QR algorithm of Francis, incorporating origin 
shifts, is one of the most satisfactory in all numerical analysis. Its properties 
and rate of convergence are particularly well understood. The following dis¬ 
cussion and related theorems define this algorithm and form the basis for its 
convergence proof. 

An n x n symmetric matrix A can initially be transformed to tridiagonal 
form by the use of (» — 2) Householder or [(n — l)(/i — 2)/2] Givens ortho¬ 
gonal similarity transformations. Thus we may assume, without loss of gen¬ 
erality, that the symmetric matrix A whose eigenvalues are to be computed 
is tridiagonal. 

The QR algorithm with origin shifts for computing the eigenvalue decom - 
position of an n x n symmetric tridiagonal matrix A can be written in the 
form 

(18.1) A, = A 

(18.2) Qk(A k — a k Q — R k 1 f , „ 

_ 1 k = 1,2 ,... 

(18.3) R k Q£ 4- a k I m = ^* + , j 

where for k— 1 , 2 ,3,..., 

(a) Q k is orthogonal 

(b) R k is upper triangular 

(c) Ok ir the kth shift parameter 

Before stating the method of computing the shift parameters a k , notation 
will be introduced for the elements of the matrices A k . It can be verified [see 
Exercise (18.46)] that A t being tridiagonal implies that all the matrices A k , 

k = 2,3.will be tridiagonal also. We shall write the diagonal terms of 

each tridiagonal matrix A k as a,'*’, i = and the superdiagonal and 

subdiagonal terms as b}*\ i = 2 ,..., n, for reference in this chapter and in 
Appendix B. 
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The shift parameters a k may now be defined as follows: 

(18.4) Each a k is the eigenvalue of the lower right 2x2 submatrix of A k , 

Ui« 

which is closest to a^ 1 . 

From Eq. (18.1) to (18.3) we have 

I = Qk^kQk 

so that all A k have the same eigenvalues. We denote them as A„ ..., A m . 

If for any k some of the b { , k) are zero, the matrix A k may be split into in¬ 
dependent submatrices in which all superdiagonal and subdiagonal elements 
are nonzero. Thus we need only give our attention to the problem for which 
all b, tky are nonzero. 

Furthermore, as is stated in Lemma (B.l) of Appendix B, all subdiagonal 
elements being nonzero implies that all eigenvalues are distinct. Thus we need 
only concern ourselves with computing eigenvalue-eigenvector decomposi¬ 
tions of symmetric tridiagonal matrices whose eigenvalues are distinct. 

The convergence of the QR algorithm is characterized by the following 
theorem of Wilkinson (1968a and 1968b). 

(18.5) Theorem {Global Quadratic Convergence of the Shifted QR Al¬ 

gorithm) 

Let A = A, be ann x n symmetric tridiagonal matrix with nonzero 
subdiagonal terms. Let matrices A k orthogonally similar to A be 
generated with the QR algorithm using origin shifts as described in 
Eq. {18.1) to {18 J) and {18.4). Then 

(a) Each A k is tridiagonal and symmetric. 

(b) The entry b?’ of A k at the intersection of the nth row and (n — l)tf 
column tends to zero as k — » oo. 

(c) The convergence ofb lk) to zero is ultimately quadratic: There exists 
e > 0, depending on A, such that for all k 

|6J* +,, | e |6i*> I 1 

The proof of this theorem is given in Appendix B. 

In practice the algorithm appears to be cubically convergent. However, 
quadratic convergence is the best that has been proved. Further remarks on 
this point are given near the end of Appendix B. 
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Section 3. COMPUTING THE SINGULAR VALUE 
DECOMPOSITION 


We now consider the construction of the singular value decomposition 
of an m X n matrix A. We assume m ;> n. See Section 1 of this chapter for 
remarks on the case m<n. 

The singular value decomposition (SVO) will be computed in two stages. 


In the first stage A is transformed to an upper bidiagonal matrix J by a 
sequence of at most 2n — 1 Householder transformations: 


(18.6) 

where 


(18.7) 


J = &(•■• ((QiA)H 2 ) ••■//„) = QPAH 


9t 



e. 


The transformation matrix Q, is selected to zero elements in rows i + 1 
through m of column /, whereas the matrix H, is selected to zero elements in 
columns / + 1 through n of row i — I. 

Note that H, is simply an identity matrix. It is included in Eq. (18.6) for 
notational convenience. Furthermore, Q„ is an identity matrix if m — n but 
is generally nontrivial if m > n. 

The second stage of the SVO computation is the application of a specially 
adapted QR algorithm to compute the singular value decomposition of B, 

(18.8) B = CsP T 


where 0 and P are orthogonal and 5 is diagonal. Then the singular value 
decomposition of A is 


(18.9) 



We now treat the computation represented by Eq. (18.8). First note that 
if any e, is zero, the matrix B separates into blocks that can be treated in- 
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dependency. Next we will show that if any q, is zero, this permits the applica¬ 
tion of certain transformations which also produce a partition of the matrix. 

Suppose that q k = 0, with q, 0, and e, =*= 0, j — k + 1,..., n. Pre¬ 
multiplying B by (it — k) Givens rotations T t we will have 


9 \ 


(18.10) T m • • ■ T k+i B = B'm 


9k -1 e k 
0 0 

?*+i **+* 



with e'j ^ 0 ,j = k + 2,..., n, and q', & 0,j = k + 

The rotation T t is constructed to zero the entry at the intersection of row 
k and column = k + 1,...,«, and is applied to rows k and j,j — k + 1, 

« • • i H • 

The matrix B' of Eq. (18.10) is of the form 

B' = 



where all diagonal and superdiagonal elements of Bf x are nonzero. Before we 
discuss #2 further, note that Bf t has at least one zero singular value since its 
lower right corner element [position (k, it)] is zero. 

When the algorithm later returns to operate on B' l this fact can be used 
to eliminate e k , with the following sequence of rotations: 


Bl = ff x R„. x • 



Here R, operates on columns / and k to zero position (/, k). For i > 1, the 
application of this rotation creates a nonzero entry in position (i — 1, k). 

We return to consideration of B’t and for convenience revert to the symbol 
B to denote this bidiagonal matrix, all of whose diagonal and superdiagonal 
elements are nonzero. The symbol n continues to denote the order of B. 

The singular value decomposition of B [Eq. (18.8)} will be obtained by an 
iterative procedure of the form 


B x — B 

B kkl = U k B k V k * = 1,2,... 
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where U k and V k are orthogonal and B k is upper bidiagonal for all k. The 
choice of U„ and V k is such that the matrix 

S — lim B k 

exists and is diagonal. 

Note that the diagonal elements of the matrix 5 which results directly 
from this iterative procedure are not generally positive or ordered. These 
conditions are obtained by appropriate postprocessing steps. 

This iterative procedure is the QR algorithm of Francis as organized for 
the singular value problem by Golub and Reinsch (1970). One step of the 
algorithm proceeds as follows. Given B k , the algorithm determines the eigen¬ 
values A ( and A 2 of the lower right 2x2 submatrix of BlB k and sets the 
shift parameter a k equal to the A, nearest the value of the lower right element 
of BlB k . 

The orthogonal matrix y k is determined so that the product 

(18.11) mB[B k - a k l .) 
is upper triangular. 

The orthogonal matrix U k is determined so that the product matrix 

(18.12) B k+t = VlB k V k 
is upper bidiagonal. 

The computational details differ from what would be suggested by the 
formulas above. In particular the matrix B k B k is never formed and the shift 
by a k is accomplished implicitly. 

To simplify the notation we shall drop the subscripts indicating the itera¬ 
tion stage, k. 

Using the notation of (18.7) the lower right 2x2 submatrix of B?B is 

Vgi-i + eJ-i "I 

U9.-1 vl + 

Its characteristic equation is 

(18.13) (?*_, + el , - AX?. 2 + el - A) - {ej m . ,) 2 = 0 

Since we seek the root of Eq. (18.13) that is closest to q* + ej, it is con¬ 
venient to make the substitution 

(18.14) d = ql + el - A 
Then d satisfies 

(18.15) 5* + (ql t-ql + el , - el)S - (e^ ,)* = 0 
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To aid in solving Eq. (18.13), define 

(18.16) / = a 

and 


(18.17) 

Then y satisfies 

(18.18) 


y = 


1 


y 2 - 2fy - 1 = 0 


The root f of Eq. (18.18) having smallest magnitude is given by 


08.19) 9 = y 

where 


(18.20) 



l~f - (l + f 2 ) 112 } if/>0 
[-/+ (1 +f 2 ) 112 ] iff < 0 


Thus, using Eq. (18.14) and (18.17), the root X of Eq. (18.13) closest to ql + 
e 1 , is 


(18.21) X = q\ + e\ - , = ql + e.(e m — 2*^') 

and the shift parameter is defined as 


(18.22) 



Next we must determine V so that V T (B r B — al) is upper triangular [see 
Eq. (18.11)]. Note that since B r B is tridiagonal, it will follow that both 
V T (B r B — al)V and V T (B r B)V are tridiagonal [see Exercise (18.46)]. 

There is a partial converse to these assertions which leads to the algorithm 
actually used to compute V. 

(18.23) Theorem [Paraphrased from Francis (1961)] 

If B T B is tridiagonal with all subdiagonal elements nonzero, V is 
orthogonal, a is an arbitrary scalar. 


(18.24) V T (B T B)V is tridiagonal 

and 


(18.2S) the first column of V T (B T B — ol) is zero below the first element, 
then V T (B I B — al) is upper triangular. 
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Using this theorem the matrices V and U of Eq. (18.11) and (18.12) will 
be computed as products of Givens rotations, 

(18.26) V = R t Rt • • • R,.i 
and 

(18.27) U T = T,. 1 -T 1 T t 


where R, operates on columns / and i + 1 of B and T, operates on rows i and 
/ -f* 1 of B. 

The first rotation R, is determined to satisfy condition (18.2S) of Theorem 
(18.23). The remaining rotations R, and T, are determined to satisfy condi¬ 
tion (18.24) without disturbing condition (18.25). 

Note that it is in the use of the matrices T, that this algorithm departs 
from the standard implicit QR algorithm for a symmetric matrix [Martin 
and Wilkinson in Wilkinson and Reinsch (1971)]. In the symmetric matrix 
problem one has a tridiagonal symmetric matrix, say, Y, and wishes to pro¬ 
duce 

(18.28) ¥ = V T YV 


In the present algorithm one has Y in the factored form B r B where B is bidi¬ 
agonal. One wishes to produce ¥ in a factored form, ¥ — B r B. Thus B must 
be of the form B = U T BV where V is the same orthogonal matrix as in Eq. 

(18.28) and U is also orthogonal. 

Turning now to the determination of the rotations R, and T, of Eq. (18.26) 
and (18.27), note that the first column of (B T B — al) is 


(18.29) 



with tr given by Eq. (18.22). The first rotation R, is determined by the require¬ 
ment that the second element in the first column of R%(B r B — of) must be 
zero. 

The remaining rotations are determined in the order T u R lt J„..., 
R,- u r._, and applied in the order indicated by the parentheses in the fol¬ 
lowing expression: 


( 18 . 30 ) 


T.-«( -mT^BR ,))*,) - R 
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The rotation T„ i — 1. n — 1, operates upon rows i and i + 1 and 

zeros the element in position (i + 1, i). The rotation R,, i = 11, 
operates on columns i and / + 1 and, for i ;> 2, zeros the element in position 
0 - 1, i + 1). 

This is an instance of an algorithmic process sometimes called chasing. 
The pattern of appearance and disappearance of elements outside the bidiago¬ 
nal positions is illustrated for the case of n = 6 by Fig. 18.1. 



Fig. 18.1 The chasing pattern for one QR sweep for the case of 
n = 6. 

From the preceding discussion it is clear that an SVD algorithm for a 
general m x n matrix can be based upon an algorithm for the more special¬ 
ized problem of the singular value decomposition of a nonsingular bidiagonal 
matrix B with elements identified as in Eq. (18.7). 

The algorithm QRBD will be stated for performing one sweep of this 
fundamental part of the SVD computation. This algorithm first tests the off- 
diagonal elements e, against a given tolerance parameter e. If | e t \ < e for any 
/, 2 i <,n, the largest index / for which | e t | < e is stored in the location 
named /, and the algorithm terminates. Otherwise the algorithm sets / — 1 to 
indicate that | e, | :> e for 2 <, i < n. The algorithm proceeds to compute the 
shift parameter [see Eq. (18.22)]. Next the algorithm determines and applies 
the rotations R, and T„ i = 1,..., n — 1 [see Eq. (18.30)]. The algorithm 
terminates with the elements of the transformed matrix B — U T BV replacing 
those of B in storage. 

For some purposes, such as solving Problem LS, it is necessary to multiply 
certain other vectors or matrices by the matrices U T and V. Algorithm QRBD 
has provision for the user to supply a k x n matrix W — {my,} and an n x p 
matrix G = {g„} which the algorithm will replace by the k x n product ma¬ 
trix WV and the n x p product matrix U T G, respectively. 
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(18.31) Algorithm QRBDfo e, n, e, /, W, k, C, p) 

Step Action 

1 Comment: Steps 2 and 3 scan for small off-diagonal ele¬ 
ments. 

2 For /:=«,« — 1,.... 2, do Step 3. 

3 If I | <, e, set / := i and go to Step 14. 

4 Set / := 1 and compute a by means of Eq. (18.16) to 
(18.22). 

5 Set / := 2. 

6 Set Ci := q, — o/q, and z e 2 [see Eq. (18.29)]. 

7 Execute Algorithm Gl(e t .„ z, c, s, e,.,). 

8 Execute Algorithm G2(c, s, q,. it e,) and for j := 1,..., k 
execute Algorithm G2(c, s, w J§l ). 

9 Set z := sq, and q, := cq,. 

10 Execute Algorithm Gl(^_,, z, c, s, q,. } ). 

11 Execute Algorithm G2(c, s, e„ q,) and for j := 1 ,...,p 
execute Algorithm G2(c, s, g,- i , J ,g,, J ). 

12 If / — n, go to Step 14. 

13 Set z := se l+u e (+I := ce <+ „ and ii + 1 and go to 
Step 7. 

14 Comment: If / = 1, one full QR sweep on the bidiagonal 
matrix has been performed. If /> 1, the element e, is 
small, so the matrix can be split at this point. 

By repeated application of Algorithm QRBD to a nonsingular upper 
bidiagonal matrix B, one constructs a sequence of bidiagonal matrices B k 
with diagonal terms q [ k) ,..., q lk) and superdiagonal terms ef\ ..., e l „ k K 
From Theorem (18.5) we know that the product e {k> q {k) converges quadra- 
tically to zero as k —► oo. The assumption that B is nonsingular implies that 
q**' is bounded away from zero. It follows that e?> converges quadratically 
to zero. In practice the iterations will be stopped when | e[ k) | < e. 

Having accepted | e ik) J as being sufficiently small, one next constructs the 
SVD of one or more bidiagonal matrices of order n — 1 or lower; this process 
is continued as above with n replaced by n — 1 until n = 1. With no more 
than n — 1 repetitions of the procedures above, the decomposition B = 0SV T 
is produced. 

With the precision parameter q in the range of about 10' 1 * to 10" • and 
e =* q 11 B 11, experience shows that generally a total of about 2 n executions of 
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Algorithm QRBD are required to reduce all superdiagonal elements to less 
than e in magnitude. 

The diagonal elements of S are not generally nonnegative or nonincreas¬ 
ing. To remedy this, choose a diagonal matrix D whose diagonal terms are 
+1 or — 1 with the signs chosen so that 

(18.32) $ = SD 

has nonnegative diagonal terms. Next choose a permutation matrix P such 
that the diagonal terms of 

(18.33) S = P T $P 
are nonincreasing. It is easily seen that 

B = {UP&P T §PX?DPf = 0S? T 

is a singular value decomposition of B with the diagonal elements of S non¬ 
negative and nonincreasing. 

The matrix V of Eq. (18.9) will be produced as a product of the form 

(18.34) V = Ht • • • HJt x • • ■ RJ)P 


where the matrices H, are the Householder transformations of Eq. (18.6), 
the matrices R, are all the postmultiplying Givens rotations produced and 
used proceeding and during QR sweeps [see Eq. (18.30) and the text following 
Eq. (18.10)], and the matrices D and P are given by Eq. (18.32) and (18.33), 
respectively. 

Similarly the matrix U T of Eq. (18.9) is generated as 


( 18 . 35 ) u r = rr M -- T X Q. • • • 0 , 

where P is given by Eq. (18.33), the rotations T, are all the premultiplying 
Givens rotations that arise proceeding and during the QR sweeps [see Eq. 
(18.10) and (18.30)], and the matrices Q, are the Householder transforma¬ 
tions of Eq. (18.6). 


Section 4. SOLVING PROBLEM L8 USING SVD 


For application to Problem LS the singular value decomposition. 



can be used to replace the problem Ax by the equivalent problem 
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(18.37) 
where 

(18.38) 
and 





b = U T b 


(18.39) x = Vp 

One may wish to consider a sequence of candidate solutions x lk} defined by 


(18.40) x ,w = 2 \ Pl v } 

where v, is the Jth column vector of V. 

Note that it is convenient to compute the vectors x (k> by the formulas 

(18.41) x <0) = 0 

(18.42) x<« = x“-» + p*t>* k = 1,..., n 

The residual norm p k associated with the candidate solution x‘*’ ( 
defined by 

p k = |Mx'*> - 6|| 

also satisfies 

(18.43) Pi- Ill’ll 1 + t W'Y 

i-T+i 

These numbers can be computed in the following manner: 

(18.44) pi — ||g*”|P 

(18.45) pi = pU, + (gStixY k = n- 1.1,0 

where g (,) and g n) are the subvectors of g defined by Eq. (18.38). 

The columns of V associated with small singular values may be inter¬ 
preted as indicating near linear dependencies among the columns of A. This 
has been noted in Chapter 12. Further remarks on the practical use of the 
singular values and the quantities p k are given in Chapter 25, Section 6. 


Section S. ORGANIZATION OF A COMPUTER 
PROGRAM FOR SVD 

We shall describe a way in which these various quantities can be computed 
with economical use of storage. This description assumes m ;> n. See Section 
1 of this chapter for remarks on the case m <n. 
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First, if mn is large and m > n, we could arrange that the data matrix 
[A : A] be first reduced to an (n + 1) upper triangular matrix by sequential 
Householder processing (see Chapter 27). The matrix A is reduced to bidi¬ 
agonal form [J] as indicated in Eq. (18.6). The nonzero elements of B re¬ 
place the corresponding elements of A in the storage array called A. The 
transformations Q, of Eq. (18.6) may be applied to b as they are formed and 
need not be saved. The vector resulting from this transformation replaces b 
in the storage array called b. The transformations H, of Eq. (18.6) must be 
saved in the storage space that becomes available in the upper triangle of the 
A array plus an n-array, called h, say. 

After the bidiagonalization the nonzero elements of B are copied into the 

two n-arrays q and e, occupying locations q it i — and e„ i — 2 . 

n, with location i, available for use as working storage by the QR algorithm. 

The computation of the matrix V of Eq. (18.36) is initiated by explicitly 
forming the product {H x • • • H m ) required in Eq. (18.34). This computation 
can be organized (and is so organized in the Fortran subroutine SVDRS of 
Appendix C) so that the resulting product matrix occupies the first n rows of 
the storage array A, and no auxiliary arrays of storage are required. 

The QR algorithm is applied to B, as represented by the data stored in 
arrays q and e. As each rotation R, is produced during the QR algorithm it 
is multiplied into the partial product matrix stored in the A array for the 
purpose of forming V [see Eq. (18.34)}. Similarly, each rotation T, is multi¬ 
plied times the vector stored in the array b in order to form the vector U T b 
[see Eq. (18.3S) and (18.38)]. 

At the termination of the QR iterations the numbers stored in the loca¬ 
tions e„ i'=2 . n will be small. The numbers stored in q„i= 1 

must next be set to be nonnegative and sorted. Application of these sign 
changes and permutations to the storage array A completes the computation 
of the matrix K[see Eq. (18.34)]. Application of the permutations to the stor¬ 
age array b completes the computation of g — l) T b [see Eq. (18.35) and 
(18.38)]. 

If desired the candidate solutions of Eq. (18.41) and (18.42) can be com¬ 
puted and stored as column vectors in the upper n x n submatrix of the 
array A, overwriting the matrix V. 

The Fortran subroutine SVDRS (Appendix C) contains the additional 
feature of initially scanning the matrix A for any entirely zero columns. If / 
such columns are found, the columns are permuted so that the first n — l 
columns are nonzero. Such a matrix has at least / of the computed singular 
values exactly zero. 

This feature was introduced into the subroutine so that a user can delete 
a variable from a problem by the simple process of zeroing the corresponding 
column of the matrix A. Without this procedure some of the computed sin- 



120 SINGULAR VALUE DECOMPOSITION AND SOLUTION OF PROBLEM L& CHAP. 18 


gular values that should be zero would have values of approximately tj\\A ||. 
In many circumstances this would be satisfactory, but in some contexts it is 
desirable to produce the exact zero singular values. 

The subroutine SVDRS also tests the rows of A, andif necessary permutes 
the rows of the augmented matrix [A : b J so that if A contains / nonzero rows 
the first fi — min (/, n) rows of the permuted matrix are nonzero. This is only 
significant when / < n, in which case A has at least n — / zero singular values 
and this process assures that at least n — / of the computed singular values 
will be exactly zero. 

When the singular value decomposition is computed for the purpose of 
reaching a better understanding of a particular least squares problem, it 
is desirable to have a program that prints various quantities derived from 
the singular value decomposition in a convenient format. Such a Fortran 
subroutine, SVA, is described and included in Appendix C. An example of 
the use of SVA is given in Chapter 26. 

EXERCISES 

(18.46) Prove that if A, is symmetric and tridiagonal, then so are the A kf 
k = 2,.... of Eq. (18.3). 

(18.47) Prove that the special QR algorithm [Eq. (18.11), (18.12), and 
(18.22)] converges in one iteration for 2 x 2 bidiagonal matrices. 
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Section 1. Normal Equations with Choleaky Decomposition 
Section 2. Modified Gram-Schmldt Orthogonallzation 


As a computational approach to least squares problems wc have stressed 
the use of Householder orthogonal transformations. This approach has the 
numerical stability characteristic of orthogonal transformations (Chapters 
15 to 17) along with ready adaptability to special requirements such as se¬ 
quential accumulation of data (see Chapter 27). We have also suggested the 
use of singular value analysis as a technique for reaching a better understand¬ 
ing of a poorly conditioned problem (Chapters 18 and 25). 

In this chapter we shall discuss some of the other computational ap¬ 
proaches to the least squares problem. For convenience Table 19.1 gives the 
high-order terms in the count of operations for the solution of the least 
squares problem A mxn x s Km > n, using various methods. 

One class of methods to be discussed is based on the mathematical equiva¬ 
lence of the least squares problem Ax ~ b and the system of linear equations 
(A T A)x = (A T b). This system is referred to as the normal equations for the 
problem. Methods based on forming and solving the normal equations typ¬ 
ically require only about half as many operations as the Householder algo¬ 
rithm, but these operations must be performed using precision q 2 to 
satisfactorily encompass the same class of problems as can be treated using 
the Householder algorithm with precision q. 

Most special processes, such as sequential accumulation of data (see 
Chapter 27), for example, can be organized to require about the same number 
of units of storage in either a normal equations algorithm or a Householder 
algorithm. Note, however, that if the units of storage are taken to be com¬ 
puter words of the same length for both methods, the Householder algorithm 
will successfully process a larger class of problems than the normal equations 
algorithm. 
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The second method to be discussed in this chapter is modified Gram- 
Schmidt orthogonalization (MGS). The numerical properties of this method 
are very similar to those of the Householder algorithm. The factored re¬ 
presentation of the Q matrix which is used in the Householder algorithm 
occupies about n 2 /2 fewer storage locations than the explicit representation 
used in MGS. Largely because of this feature, we have found the Householder 
algorithm more conveniently adaptable than MGS to various special ap¬ 
plications. 


Table 19.1 operation counts for various least squares 

COMPUTATIONAL METHODS 


Method 

Asymptotic Number of 
Operations Where an Operation 

Is a Multiply or Divide Plus 
an Add 

Householder triangularization 

Singular value analysis 

mn 2 - *V3 

Direct application to A 

Householder reduction of A to triangular R 

2m* 2 4 <r(*)t 

plus singular value analysis of R 

m« 2 4- 5* 3 /3 + *(«)t 

Form normal equations 

m* 2 /2 

Choksky solution of normal equations 
Gauss-Jordan solution of normal equations 

*3/6 

(for stepwise regression) 

*3/3 

Eigenvalue analysis of normal equations 

4*3/3 4 <r(*)t 

Gram-Schmidt (either classical or modified) 

mn 2 


tThe term a(n) accounts for the iterative phase of the singular value or eigenvalue com¬ 
putation. Assuming convergence of the QR algorithm in about 2 k sweeps and implementa¬ 
tion as in the Fortran subroutine ORBD (Appendix C), o(n) would be approximately 4k 3 . 


As a specific example of this compact storage feature of the Householder 
transformation, note that one can compute the solution X of Ax s b and the 
residual vector r = b — AX using a total of m(n + 1) -f 2 n + m storage 
locations. Either the use of normal equations or the MGS algorithm requires 
an additional n(n + l)/2 storage locations. Furthermore, as mentioned be¬ 
fore, the assessment of storage requirements must take into consideration the 
fact that the normal equations method will require ^-precision storage loca¬ 
tions to handle the class of problems that can be handled by Householder or 
MGS using ^-precision storage locations. 


Section 1. NORMAL EQUATIONS WITH CHOLESKY 
DECOMPOSITION 

The method most commonly discussed in books giving brief treatment of 
least squares computation is the use of normal equations. Given the problem 
A mxm x ~ b one premultiplies by A T , obtaining the system 
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(19.1) P mXm x = rf„ xl 

which is called the system of normal eqtuttions for the problem. Here 

(19.2) P = jFA 

and 

(19.3) d = A T b 

Equation (19.1) arises directly from the condition that the residual vector 
b — Ax for the least squares solution must be orthogonal to the column 
space of A. This condition is expressed by the equation 

(19.4) A T (b - Ax) = 0 

which is seen to be equivalent to Eq. (19.1) using Eq. (19.2) and (19.3). 

While P has the same rank as A and thus can be singular, it is nevertheless 
true that Eq. (19.1) is always consistent. To verify this note that, from Eq. 
(19.3), the vector d is in the row space of A but from Eq. (19.2), the row space 
of A is identical with the column space of P. 

The system (19.1) could be solved by any method that purports to solve 
a square consistent system of linear equations. From Eq. (19.2), however, 
the matrix P is symmetric and nonnegative definite, which permits the use of 
Cholesky elimination with its excellent properties of economy and stability 
[Wilkinson (1965a), pp. 231-232]. 

The Cholesky method, which we will now describe, is based on the fact 
that there exists an n x n upper triangular (real) matrix U such that 

(19.5) U T U = P 

and thus the solution x of Eq. (19.1) can be obtained by solving the two tri¬ 
angular systems 

(19.6) V T y - d 

and 

(19.7) Ux = y 

Alternatively, the process of solving Eq. (19.6) for y can be accomplished 
as a part of the Cholesky decomposition of an appropriate augmented ma¬ 
trix. For this purpose define 

J = [A:b] 


and 

(19.8) 


\_d T ca I J 
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Note that P and d of Eq. (19.8) denote the same matrix and vector as in Eq 
(19.2) and (19.3) and co satisfies 

CD 1 = b T b 


The upper triangular Cholesky factorization of P is then of the form 


(19.9) 

II 

with 


(19.10) 

;] 


n 1 


where U and y of Eq. (19.10) satisfy Eq. (19.5) and (19.6). Furthermore it is 
easily verified that p of Eq. (19.10) satisfies 

|p| = ||*-^|| 

where X is any vector minimizing \\b — Ax\\. 

For convenience the details of computing the Cholesky decomposition 
will be described using the notation of Eq. (19.5). Obviously the algorithm 
also applies directly to Eq. (19.9). 

From Eq. (19.5) we obtain the equations 

(19.11) u kl u kJ =p„ for i = 1. n;j = /,..., n 

Solving for u,j in the equation involving p u leads to the following equa¬ 
tions, which constitute the Cholesky (also called square root or Banachiewicz) 
factorization algorithm: 


(19.12) 

(19.13) 

(19.14) 





Pu — ^ V kt U k ) 


«u 


j = i+ 1... 




n 


In Eq. (19.12) and (19.14) the summation term is taken to be zero when 
i=l. 

Theoretically if Rank (A) — n, then v, > 0, / = 1,...,«. Equations 
(19.12) to (19.14) then define unique values for each u,,. 

If Rank (A) — k <n however, there will be a first value of /, say, i = t, 
such that v, = 0. Then for i = t the numerator in the right side of Eq. (19.14) 
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must also be zero for/ = / + 1. n, since solutions exist for all the equa¬ 

tions in (19.11). In this case there is some freedom in the assignment of values 
to u„J — t + 1,..., n. One set of values that is always admissible is u tJ = 
0,j = t -f 1,. .., n [see Exercise (19.37)]. 

It follows that the theoretical algorithm of Eq. (19.12) to (19.14) provides 
a solution for Eq. (19.5) regardless of the rank of A if it is modified so that 
Eq. (19.14) is replaced by 

(19.15) u,j — 0 j=*i+ 


for any value of / for which u„ — 0. 

Note that the upper triangular matrix R of the Householder decomposi¬ 
tion QA = satisfies R T R — P. If Rank (A) — n, the solution of Eq. (19.5) 

is unique to within the signs of rows of V [see Exercise (2.17)]. Thus if Rank 
(A) — n, the matrix R of the Householder decomposition is identical with 
the Cholesky matrix V to within signs of rows. 

The Cholesky decomposition can also be computed in the form 


(19.16) 


L T L = P 


where L is an n x n lower triangular matrix. The formulas for elements of L 
are 



In actual computation there arises the possibility that the value of v, 
computed using Eq. (19.12) for the Cholesky decomposition of the (theoret¬ 
ically) nonnegative definite matrix P of Eq. (19.2) may be negative due to 
round-off error. Such errors can arise cumulatively in the computation as¬ 
sociated with Eq. (19,2), (19.12), (19.13), and (19.14). 

If a computed v t is negative for some /, one method of continuing the 
computation is to set v, = 0, u„ — 0, and apply Eq. (19.15). A Fortran sub¬ 
routine using essentially this idea is given by Healy (1968). 

Another approach is to perform and record a symmetric interchange of 
rows and columns to maximize the value of v, at the /th step of the algorithm. 
This has the effect of postponing the occurrence of nonpositive values of v„ 
if any, until all positive values have been processed. When all remaining 
values of v, are nonpositive, then the corresponding remaining rows of V can 
be set to zero. 
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The implementation of such an interchange strategy requires some reor¬ 
dering of the operations expressed in Eq. (19.12) to (19.14) so that partially 
computed values of v, will be available when needed to make the choice of 
pivital element. When interchanges are used on an augmented matrix of the 
form occurring in Eq. (19.8), the last column and row must not take part in 
the interchanges. 

A Cholesky decomposition for P of Eq. (19.8) gives rise to an upper 
triangular matrix 0 of Eq. (19.10) having the same relationship to Problem 
LS as does the triangular matrix obtained by Householder triangularization 
of A. This can be a useful observation in case one already has data in the 
form of normal equations and then wishes to obtain a singular value analysis 
of the problem. 

Ignoring the effects of round-off error, the same information obtainable 
from singular value analysis of A (see Chapter 18 and Chapter 25, Section 5) 
can be obtained from an eigenvalue analysis of the matrix P of Eq. (19.2). 
Thus if the eigenvalue-eigenvector decomposition of P is 

P = v&V T 

with eigenvalues, sf i ^ r| 2 ;> • • • > si n , one can compute 

g'» = 

where d and aP are defined by Eq. (19.8). Then with the change of variables 

x = Vp 


Problem LS is equivalent to the problem 


(19.19) 



This least squares problem is equivalent to problem (18.37) obtained from 
singular value analysis since the diagonal matrix 5 and the n-vector g (I> of 
Eq. (19.19) are the same as in Eq. (18.37) and y of Eq. (19.19) and g ,2> of Eq. 

(18.37) satisfy y = || g'** |l- 

For a given arithmetic precision tf, however, the quantities S,g <l \ and 
y — ||g (2> II will be determined more accurately by singular value analysis 
of [A : b] or of the triangular matrix R = Q[A : b] obtained by Householder 
triangularization than by an eigenvalue analysis of [A : b] T [A : b]. 

As mentioned in the introduction of this chapter, using arithmetic of a 
fixed relative precision tf, a larger class of problems can be solved using 
Householder transformations than can be solved by forming normal equa- 
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tions and using the Cholesky decomposition. This can be illustrated by con¬ 
sidering the following example. 

Define the 3 x 2 matrix 


A = 


1 1 

1 1 

1 1 - f 


Suppose the value of t is such that it is significant to the problem, e > 100i/, 
say, but e 1 < tf, so that when computing with relative precision tj we have 
1 - f # 1 but 3 4- e 1 is computed as 3. Thus instead of computing 


jfaJ 3 3 ~ f 1 

[_3 — e 3-2e + e\ 


we shall compute the matrix 


r 3 3-0 

1.3-f 3-2(1 


plus some random errors of order of magnitude 3/f. 
Further computation of the upper triangular matrix 


= j >u r,i '1 

Lo r„J 


that satisfies /FA = A T A yields, using the Cholesky method [Eq. (19.12) to 
(19.14)], 

r 11 

_ 3 — e 

r,1 ~VT 

r it — Pn ~ r ti 

This final equation is computed as 

rh = 3-2e- = 0 

plus random errors of the order of magnitude 3tf and of arbitrary sign. The 
correct result at this point would be of course 


rfi = 3 — 2f + e* — 


9 — 6e + e 2 2e* 
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Thus using arithmetic of relative precision 9 , no significant digits are ob¬ 
tained in the element r 2J . Consequently the matrix R is not distinguishable 
from a singular matrix. This difficulty is not a defect of the Cholesky decom¬ 
position algorithm but rather reflects the fact that the column vectors of A*A 
are so nearly parallel (linearly dependent) that the fact that they are not 
parallel cannot be established using arithmetic of relative precision 9 . 

Contrast this situation with the use of Householder transformations to 
triangularize A directly without forming A T A. Using Eq. (10.5) to (10.11) one 
would compute 

s — —a/T 

1 +/T 

u — 1 

1 

b - -^(1+^/7) = -(3 + ,/T) 

Then to pramultiply the second column, a lt of A by / + b~ 1 utf one computes 

t — b~ l (u T a t ) = 
and 





The important point to note is that the second and third components of 
which are of order of magnitude f, were computed as the difference of 
quantities of order of magnitude unity. Thus, using (/-precision arithmetic, 
these components are not lost in the round-off error. 

The final step of Householder triangularization would be to replace the 
second components of S 2 by the negative euclidean norm of the second and 
third components, and this step involves no numerical difficulties. 





3 
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Collecting results we obtain 



plus absolute errors of the order of magnitude 3tf. 

In concluding this example we note that, using q-precision arithmetic, 
applying Householder transformations directly to A produces a triangular 
matrix that is clearly nonsingular, whereas the formation of normal equa¬ 
tions followed by Cholesky decomposition does not. 


Section 2. MODIFIED GRAM-SCHMIDT 
ORTHOGONAUZATION 


Gram-Schmidt orthogonalization is a classical mathematical technique 
for solving the following problem: Given a linearly independent set of vec¬ 
tors [a u ..., a„}, produce a mutually orthogonal set of vectors {q u ..., 
such, that for k — 1 ,...,», the set [q u ..., q*} spans the same ft-dimen- 
sional subspace as the set ..., a*}. The classical set of mathematical for¬ 
mulas expressing q t in terms of a, and the previously determined vectors 
q u ..., q } .i appear as follows: 

(19.20) q t = a t 

(19.21) q, = a, - r tJ q, j = 2,..., n 
where 


(19.22) 



To convert to matrix notation, define A to be the matrix with columns 
a Jt Q to be the matrix with columns q„ and JR to be the upper triangular 
matrix with unit diagonal elements with the strictly upper triangular elements 
given by Eq. (19.22). Then Eq. (19.20) and (19.21) can be written as 


(19.23) A=*QR 

It is apparent that Gram-Schmidt orthogonalization can be regarded as being 
another method for decomposing a matrix into the product of a matrix with 
orthogonal columns and a triangular matrix. 

Experimental evidence [Rice (1966)] indicated that Eq. (19.20) to (19.22) 
have significantly less numerical stability than a certain mathematically 
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equivalent variant. The stability of this modified Gram-Schmidt method was 
established by Bjbrck (1967a), who obtained the expressions given below as 
Eq. (19.35) and (19.36). 

Note that the value of the inner product (ctfq,) will not change if a, is 
replaced by any vector of the form 

(19.24) a, - ^ <t k q k 

since q T k q, — 0 for k ^ i. In particular, it is recommended that Eq. (19.22) 
be replaced by 

< 19 ' 25 > “ $5? 

where 

(19.26) af = a y — r kj q k 

In fact, if one poses the problem of choosing the numbers a* to minimize the 
norm of the vector defined by expression (19.24), it is easily verified that the 
minimal length vector is given by Eq. (19.26). Thus the vector a y of Eq. (19.22) 
and the vector af of Eq. (19.25) are related by the inequality 11 oj, 0 11 <; ||a y ||. 
The superior numerical stability of this modified algorithm derives from this 
fact. 

The amount of computation and storage required is not increased by this 
modification since the vectors a ) 0 can be computed recursively rather than 
using Eq. (19.26) explicitly. Furthermore, the vector a l j* n can replace the 
vector af in storage. 

This algorithm, which has become known as the modified Gram-Schmidt 
(MGS) algorithm [Rice (1966)], is described by the following equations: 


(19.27) 

= fly 

J — If • • • $ ft 

(19.28) 

q, = af 

* 

(19.29) 

df = qjq, 


(19.30) 

1 ! 

j — i + 1. n 

(19.31) 

<■> = af - r tl q, ) 

. 


To use MGS in the solution of Problem LS one can form the augmented 
matrix 

(19.32) A = [A: b] 

and apply MGS to the mx(«+l) matrix A to obtain 

(19.33) A = Q& 
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where the matrix R is upper triangular with unit diagonal elements. The 
strictly upper triangular elements of R are given by Eq. (19.30). The vectors 
q, given by Eq. (19.28) constitute the column vectors of the mx(«+l) 
matrix Q. One also obtains the (n + 1) x (n + 1) diagonal matrix 3 with 
diagonal elements d„ i = 1,..., n -4- 1, given by Eq. (19.29). 

For the purpose of mathematical discussion leading to the solution of 
Problem LS, let Q be an m x m orthogonal matrix satisfying 


Then 

(19.34) 


Q m 


' 3 l}n + l 

0 ]}m — n 
n+ 1 



»x(n+l) 



Dc' 

^«+j 

0 


where 3 and R have been partitioned as 



Then for arbitrary x we have 

\\Ax — 6||* = WfFiAx - 4)||* 

= ||/>(** - <0II* + . 

Therefore the minimum value of the quantity ||^4x — b\\ is \d H+x \ and this 
value is attained by the vector St, which satisfies the triangular system 

Rx — c 

For a given precision of arithmetic MGS has about the same numerical 
stability as Householder triangularization. The error analysis given in Bjdrck 
(1967a) obtains the result that the solution to Ax s b computed using MGS 
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with mixed precision (g, tf 1 ) is the exact solution to a perturbed problem 
(A + E)x=zb +/with 

(19.35) UEWr^W'WAWrn 
and 

(19.36) ||/||<;2»*'»|I*IIV 

Published tests of computer programs [e.g., Wampler (1969)] have found 
MOS and Householder programs to be of essentially equivalent accuracy. 

The number of arithmetic operations required is somewhat larger for 
MOS than for Householder (see Table 19.1) because in MGS one is always 
dealing with column vectors of length m, whereas in Householder triangu- 
larization one deals with successively shorter columns. This fact also means 
that programs for MOS usually require more storage than Householder 
programs, since it is not as convenient in MGS to produce the R matrix in 
the storage initially occupied by the matrix A. With some extra work of mov¬ 
ing quantities in storage, however, MOS can be organised to produce Jf in 
the storage initially occupied by A if one is not saving the 9 , vectors. 

The MGS method can be organized to accumulate rows or blocks of rows 
of [A : 6] sequentially to handle cases in which the product mn is very large 
and m > n. This possibility rests on the fact that the (« + 1) x (» + 1) 
matrix 0R) [see Eq. (19.34)] defines the same least squares problem as the 
m x (n + 1) matrix [A : b ]. This fact can be used to develop a sequential 
MGS algorithm in the same way as the corresponding property of the 
Householder triangularization is used in Chapter 27 to develop a sequential 
Householder method. 

Specialized adaptations of Gram-Schmidt orthogonalization include the 
conjugate gradient method of solving a positive definite set of linear equations 
[Hestenes and Stiefel (1952); Beckman, appearing in pp. 62-72 of Ralston 
and Wilf (1960); Kammerer and Nashed (1972)] that has some attractive 
features for large sparse problems [Reid (1970), (1971a), and (1971b)] and 
the method of Forsythe (1957) for polynomial curve fitting using orthogo- 
nalized polynomials. 


EXERCI8E8 

(19.37) Let R be an n x n upper triangular matrix with some diagonal 
elements r„ = 0. Show that there exists an n x n upper triangular 
matrix U such that IFU = JCR and u lf = 0,j — i,... ,n, for all 
values of i for which r„ = 0. Hint: Construct U in the form U = 
QR, where Q is the product of a finite sequence of appropriately 
chosen Givens rotation matrices. 
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(19.38) [Peters and Wilkinson (1970)] Assume Rank (A mxx ) — n and m > n. 
By Gaussian elimination (assuming partial pivoting) the decomposi¬ 
tion A = PLR can be obtained where L mxx is lower triangular with 
unit diagonal elements, R, xa is upper triangular, and P is a per¬ 
mutation matrix accounting for the row interchanges. 

(a) Count operations needed to compute this decomposition. 

(b) Problem LS can be solved by solving ULy = L T P T b by the 
Cholesky method and then solving Rx — y. Count the oper¬ 
ations needed to form UL and to compute its Cholesky 
factorization. 

(c) Show that the total operation count of this method is the same 
as for the Householder method (see Table 19.1). (We suggest 
determining only the coefficients of mn 1 and it 3 in these opera¬ 
tion counts.) 


(19.39) 


Let Abeanm x n matrix of rank n. Let A 




be the decom¬ 


position of A obtained by the (exact) application of the modified 
Gram-Schmidt algorithm to A. Assume Q and R are adjusted so 
that the columns of Q have unit euclidean length. Consider the 
(exact) application of the algorithm HFT (11.4) to the (m + it) x n 


matrix 



and let 



denote the data that replace 




storage after execution of HFT. Show that B is the same as R 
to within signs of rows and C is the same as Q to within signs 

and normalization of columns. 


(19.40) ( Cholesky Method Without Square Roots) In place of Eq. (19.5) 
consider the decomposition R^DW — P, where W is upper trian¬ 
gular with unit diagonal elements and d is diagonal. Derive 
formulas analogous to Eq. (19.12) to (19.14) for computing the 
diagonal elements of D and the strictly upper triangular elements 
of W. Show that using this decomposition of P [Eq. (19.8)] prob¬ 
lem (19.1) can be solved without computing any square roots. 


LINEAR LEAST SQUARES WITH 
OO LINEAR EQUALITY CONSTRAINTS 

using a basis of the null space 


In this chapter we begin the consideration of least squares problems in 
which the variables are required to satisfy specified linear equality or inequal¬ 
ity constraints. Such problems arise in a variety of applications. For example, 
in fitting curves to data, equality constraints may arise from the need to inter¬ 
polate some data or from a requirement for adjacent fitted curves to match 
with continuity of the curves and possibly of some derivatives. Inequality 
constraints may arise from requirements such as positivity, monotonicity, 
and convexity. 

We shall refer to the linear equality constrained least squares problem as 
Problem LSE. The problem with linear inequality, as well as possibly linear 
equality constraints, will be called Problem LSI. Three distinct algorithms 
for solving Problem LSE will be given here and in Chapters 21 and 22, 
respectively. Problem LSI will be treated in Chapter 23. 

To establish our notation we state Problem LSE as follows: 

(20.1) Problem LSE ( Least Squares with Equality Constraints ) 

Given an m, xn matrix C of rank k,, an m,-vector d, mm, xn 
matrix E, and an m 4 -vector f, among all n-vectors x that satisfy 

(20.2) Cx = d 
find one that minimizes 

(20.3) \\Ex- /|| 

Clearly Problem LSE has a solution if and only if Eq. (20.2) is consistent. 
We shall assume the consistency of Eq. (20.2) throughout Chapters 20, 21, 
22, and 23. In the usual practical cases of this problem one would have 
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n > m, —k„ which would assure that Eq. (20.2) is consistent and has more 
than one solution. 

It will subsequently be shown that, if a solution for Problem LSE exists, 
it is unique if and only if the augmented matrix is of rank n. In the case 

of nonuniqueness there is a unique solution of minimal length. 

Clearly Problem LSE could be generalized to the case in which Eq. (20.2) 
is inconsistent but is interpreted in a least squares sense. We have not seen this 
situation arise in practice; however, our discussion of Problem LSE would 
apply to that case with little modification. 

The three algorithms to be given for Problem LSE are each compact in 
the sense that no two-dimensional arrays of computer storage are needed 
beyond that needed to store the problem data. Each of the three methods can 
be interpreted as having three stages: 

1. Derive a lower-dimensional unconstrained least squares problem from 
the given problem. 

2. Solve the derived problem. 

3. Transform the solution of the derived problem to obtain the solution 
of the original constrained problem. 

The first method [Hanson and Lawson (1969)], which will be described in 
this chapter, makes use of an orthogonal basis for the null space of the matrix 
of the constraint equations. This method uses both postmultiplication and 

premultiplication of the given matrix by orthogonal transformation 

If the problem does not have a unique solution, this method has the 
property that the (unique) minimum length solution of the derived uncon¬ 
strained problem gives rise to the (unique) minimum length solution of the 
original constrained problem. Thus this first method is suggested for use 
if it is expected that the problem may have deficient pseudorank and if stabi¬ 
lization methods (see Chapter 25) based on the notion of limiting the length 
of the solution vector are to be used. 

This method is amenable to numerically stable updating techniques for 
solving a sequence of LSE problems in which the matrix C is successively 
changed by adding or deleting rows. The method is used in this way by Stoer 
(1971) in his algorithm for Problem LSI. 

Furthermore, this first method is of theoretical interest. In Theorem 
(20.31) this method is used to show the existence of an unconstrained least 
squares problem which is equivalent to Problem LSE in the sense of having 
the same set of solutions for any given right-side vectors d and f This permits 
the application to Problem LSE of certain computational procedures, such 
as singular value analysis. 
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The second method, presented in Chapter 21, uses direct elimination by 
premultiplication using both orthogonal and nonorthogonal transformation 
matrices. 

Either of the first two methods is suitable for use in removing equality 
constraints (if any) as a first step in solving Problem LSI. Also, either method 
is adaptable to sequential data processing in which new rows of data are 
adjoined to either the constraint system [C: d] or the least squares system 
[E: f] after the original data have already been trianguiarized. 

The third method for Problem LSE, presented in Chapter 22, illuminates 
some significant properties of the Householder transformation applied to 
a least squares problem with disparately weighted rows. From a practical 
point of view the main value of this third method is the fact that it provides 
a way of solving Problem LSE in the event that one has access to a subroutine 
implementing the Householder algorithm for the unconstrained least squares 
problem but one does not have and does not choose to produce code imple¬ 
menting one of the specific algorithms for Problem LSE. 

We turn now to our first method for Problem LSE. This method makes 
use of an explicit parametric representation of the members of the linear flat 

(20.4) X = {x:Cx — d] 

If 

(20.5) C = HRK T 

is any orthogonal decomposition (see Chapter 2 for definition) of C, and K is 
partitioned as 

(20.6) K - [£, £*]}/i 

k x n — k x 

then from Theorem (2.3), Eq. (2.8), it follows that X may be represented as 

(20.7) X={x:x = x + 
where 

(20.8) x = C+d 

and y t ranges over the space of all vectors of dimension n — k x . 

(20.9) Theorem 

Assuming Eq. (20.2) is consistent. Problem LSE has a unique minimal 
tength solution given by 
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( 20 . 10 ) 


ft = C+d + (EZ) + (f - EC+d) 


where 


Z = l n -C + C 


or equivalently 


( 20 . 11 ) 


ft = CV + *,(£*,)+(/ - EC*d ) 

where K 2 is defined by Eq. (20.6). 

The vector ft is the unique solution vector for Problem LSE if and 

only if Eq. (20.2) is consistent and the rank o/j^j is n. 


Proof: To establish the equivalence of Eq. (20.10) and (20.11) note 
first that 

Z = J, - C+C = KK T - KR+RK r 
= KK T - /'*■ °1 K r = KK r -K i K\ 

Lo Oj 

= K a Kl 


and thus 


The fact that 


EZ = EK»Kl 


(EKiKD* = K t (EK t y 


can be verified by directly checking the four Penrose conditions [Theorem 
(7.9)] and recalling that 

KIKi - 


Thus Eq. (20.10) and (20.11) are equivalent. 

Using Eq. (20.7) it is seen that the problem of minimizing the expression 
of Eq. (20.3) over all x € X is equivalent to finding an (n — A:,)-vector y 2 that 
minimizes 

II E[x + Kiy%) — / |I 

or equivalently 

(20.12) \\(EKi)y 2 - (f - Ex)\\ 

From Eq. (7.5) the unique minimal length solution vector for this problem 
is given by 


( 20 . 13 ) 


9* = (EK$)*(f — Ex) 
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Thus using Eq. (20.7) a solution vector for Problem LSE is given by 

(20.14) St = jc + K t p t 

which is equivalent to Eq. (20.11) and thus to Eq. (20.10) also. 

Since the column vectors of K t are mutually orthogonal, and also orthog¬ 
onal to x, the norm of any vector x e X satisfies 

(20.15) IOT HOT+ 11*11* 

If ft 9 * pi also minimizes Eq. (20.10), then ||y 2 || > || 1|. Thus xx + 

Kipi is a solution to LSE but 

(20.16) |OT = 11*11* + liftII* > Hill* + llftll* = Hill* 


It follows that Jc is the unique minimum length solution vector for Problem 
LSE. 

It remains to relate the uniqueness of the solution of Problem LSE to 
k — Rank (f^TIY Clearly if k < n there exists an n-vector w ^ 0 satisfying 


(20.17) 



w = 0 


and if St is a solution of Problem LSE, so is £ + w. 

On the other hand, if k — n, consider the (m, 4- m 2 ) x n matrix 


(20.18) 



whose rank is also n. Having only n columns, all columns must be indepen¬ 
dent. Since CK Z = 0, it follows that EK Z must be of rank n — k x . Thus)) of 
Eq. (20.13) is the unique vector minimizing Eq. (20.12), and consequently St 
of Eq. (20.14) is the unique solution of Problem LSE. 

This completes the proof of Theorem (20.9). 


A computational procedure can be based on Eq. (20.11). We shall assume 
k t — m t since this is the usual case in practice. Then the matrix H of Eq. 
(20.5) can be taken as the identity matrix. 

Let K be an n x n orthogonal matrix that when postmultiplied times C 
transforms C to a lower triangular matrix. Postmultipiy C and E by K; thus 
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(20.19) 


[M 1 il: 


m i /i — #«! 

where C, is m, x m, lower triangular and nonsingular. 

Solve the lower triangular system 

(20.20) d?,y, = d 
for the m,-vector _p,. Compute 

(20.21) / = / - £j>, 

Solve the least squares problem 

(20.22) £ iyi s / 

for the (n — w,>vector _p 2 . Finally, compute the solution vector, 


(20.23) 



The actual computational steps are described by means of the following 
algorithm. Initially the matrices and vectors C, d, E, and f are stored in arrays 
of the same names. The array g is of length m t . The arrays h, u, and p are each 
of length n — mi. The solution of minimal length is returned in the storage 
array, x, of length n. The parameter t is a user provided tolerance parameter 
needed at Step 6. 

(20.24) Algorithm LSE(C, d, E,f, m u m lt n, g, k, u,p, x, t) 

Step Description 

1 For i 1,..., m u execute Algorithm Hl’(/, i + 1, n, c n , 
g t >Ci +—/). [See Eq. (20.19). Here and in Step 2 
Algorithms HI and H2 operate on rows of the arrays C 
and E.] 

2 For i1,..., m u execute Algorithm H2(i, i + 1, n, c„, 
gt> e ii, m i). [See Eq. (20.19). If the matrices C and E were 
stored in a single (m, + m t ) x n array, say W, then Steps 

1 and 2 could be combined as: For / := 1.m, execute 

Algorithm Hl(i, i + 1,», w,„ g„ w (+)iI , /»,+»»*- /).] 

3 Set := djc u . If m t = 1, go to Step 5. 
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Step Description 

4 For i 2,..., set 

x, := ^ c lt x^jc„. 

[See Eq. (20.20).] 

5 For / := 1,....set 

// * = /r — ^Lj 

[See Eq. (20.21).] 

6 Execute Algorithm HFTI(ei trni+l , m 2 , n - mi, f, r, x mi+1 , 
k, h, u, p) of Eq. (14.9). [The solution y 2 of Eq. (20.22) 
has been computed and stored in locations x<, * := mi+1, 

. . . j 71 . j 

7 For i := «i„ m, — 1.1, execute Algorithm H2(/, 

i + 1, n, c, j, g„ x, 1). [See Eq. (20.23). Here Algorithm H2 
refers to rows of the array C and applies transformations to 
the singly subscripted array, x.] 

8 Comment: The array named x now contains the minimal 
length solution of Problem LSE. 


As an example of Problem LSE, consider the minimization of 11 Ex — f || 
subject to Cx — d where 


(20.25) 

(20.26) 

(20.27) 

(20.28) 


C = [0.4087 0.1593] 
_ T0.4302 0.35161 
[.0.6246 0.3384J 
«/= 0.1376 



The Householder orthogonal matrix, K, which triangularises C from 
the right is 


•-0.9317 -0.3632' 
.-0.3632 0.9317. 


Of course this matrix would not usually be computed explicitly in executing 
Algorithm LSE (20.24). Using Eq. (20.19) to (20.23) we compute 
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(20.29) 


rffl 

01 

-0.4386 

0.0 ' 

\m 

E 1 

-0.5285 

0.1714 

L"t 

^aJ 

.-0.7049 

0.0885. 


A 0.1376 f\414>T 

9t =04356 0J137 


, = j"0.49351 

f ~ L0.7455J 

$ 2 = 4.0472 

*=r- U775 i 

L 3.8848J 


We turn now to some theoretical consequences of Theorem (20.9). Equa¬ 
tion (20.10) can be rewritten as 


if we define 
(20.30) 

and 


St = A+b 


A* = [C*~ (EZYEC +: (£Z) + ] 

= [C* - K 2 (EK 2 yEC *: K 2 (EK 2 )*] 


But Eq. (20.30) implies that X is the unique minimum length solution to 
the problem of minimizing 


\\Ax — b\\ 


where A is the pseudoinverse of the matrix A* defined in Eq. (20.30). 
(20.31) Theorem 

The pseudoinverse of the matrix A + defined in Eq. (20.30) is 


(20.32) 

where 



(20.33) i = (EZXEZ) + E = E(EZ) + E = (EK 2 XEK 2 )*E 


and Z and Kj are defined as in Theorem (20.9). 
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Proof: Recall that 

CK 2 = 0 
K\C+ = 0 
K r 2 K 2 = I m . m 

Z = I. ~ C+C — K 2 K T 2 
and 

(EZ) + = {EK 2 K$+ = * 2 (£**) + 

Other relations that follow directly from these and will be used in the 
proof are 

EZC+ = EKtKlC+ = 0 
C(EZ)+ = CK 2 (EK 2 )+ = 0 
and 


(20.34) 


EHEZy = EK 2 (EK 2 )+ 

= EK 2 K\K 2 (EK 2 y 
= {EZ%EZ)+ 


Equation (20.34) establishes the equality of the different expressions for £ in 
Eq. (20.33). 

Next define 


-c 


L(£ZX£Z) + £J 

and verify the four Penrose equations [Theorem (7.9)] as follows: 

A+X = C+C - (. EZ)+EC+C + {EZy(EZ){EZ)*E 
= C+C - (EZ)+E(C+C - /) 

= C+C + (EZ)+(EZ) 


which is symmetric. 


XA+ = 


CC+ - C(EZ)+EC+ 

(J EZXEZ)+E[l - (EZ)+E)C+ 

CC+ 0 
0 (EZfEZ)+_ 


C(EZ)+ 

(EZXEZ)+E(EZ)+_ 


which is symmetric. 
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A+XA + = [C + C + ( EZ)+(EZ)HC* - (EZ)*EC* :(£Z) + ] 

= {C+ - (EZ)+(EZftI - (£Z) + £]C + : (£Z) + (£Z)(£Z) + } 
= [C+ - (£Z) + (£Z)(£Z) + £C + : (£Z) + ] 

= [C + - (EZ) + EC + : (EZ)*] = 


jM + jr = 


c 

_(£Z)(£Z) + £_ 


[C+C + (£Z)+(£Z)] 


-[ 

-C 


(EZ)(EZ)*EC*C + (£Z)(£Z) + £(£Z) + £Z 

c 1 r C 


] 


{EZXEZyE[C+C + Z]J L(^^X^Z) 




+ £ 



This completes the proof of Theorem (20.31). 


As a numerical illustration of Theorem (20.31) we evaluate Eq. (20.32) 
and (20.33) for the data (20.25) to (20.28). From Eq. (20.33) we have 




-c 


(JBKMBK&E 

r°' 17l4 l[4.6076 2.3787] 
1.0.0885J L 

0.5943 0.4155"! 

0.3068 0.2145J 


0.4302 

0.6246 


and from Eq. (20.32) 


-ra- 


0.4087 0.1593 
0.5943 0.4155 
0.3068 0.2145 


0.3516"| 

0.3384J 


According to Theorem (20.31) this matrix has the property that for an arbi¬ 
trary one-dimensional vector, d, and two-dimensional vector,/, the solution 
of the least squares problem 



is also the solution of the following Problem LSE: Minimize ||Zjc — /|| 
subject to Cx — d where C and E are defined by Eq. (20.25) and (20.26). 

It should be noted, however, that the residual norms for these two least 
squares problems are not in general the same. 
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JM LINEAR EQUALITY CONSTRAINTS 
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In this chapter Problem LSE, which was introduced in the preceding 
chapter, will be treated using a method of direct elimination by premultiplica¬ 
tion. It will be assumed that the rank k, of C is m t and the rank of is n. 

From Theorem (20.9) this assures that Problem LSE has a unique solution for 
any right-side vectors d and /. 

It will be further assumed that the columns of are ordered so that 

the first m t columns of C are linearly independent. Column interchanges to 
achieve such an ordering would be a necessary part of any computational pro¬ 
cedure of the type to be discussed in this chapter. 

Introduce the partitioning 



The constraint equation Cx = d can be solved for x, and will give 


x, = CT'(d - CixJ 


Substituting this expression for x, in \\Ex —/1[ gives 
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(21.3) \\Ex - /|| - \\E t CV{d - C 2 x 2 ) + E 2 x 2 - f || 

= \\(E 2 - B&i'Cjxt -(f- E x CVd)\\ 

= \\£l*l - /II 

as the expression to be minimized to determine x 2 . 

Conceptually this leads to the following solution procedure: Compute 

(21.4) £ 2 = E 2 ~ £,Ct‘C, 
and 

(21.5) / = / - EiCj'd 
Solve the least squares problem 

(21.6) £ 2 x 2 s / 

and finally compute 

(21.7) Xi =C- x \d-C 2 x 2 ) 

There are a variety of ways in which these steps can be accomplished. For 
example, following BjOrck and Golub (1967), suppose a QR decomposition 
of C t is computed so that 

(21.8) c, - ere, 

where Q t is orthogonal and £, is upper triangular. Then Eq. (21.4) and (21.5) 
can be written as 

(21.9) £ 2 = e 2 - (sA’Xe.c,) sE 2 - £A 
and 

(21.10) / = /- (E& 'XS,d) = f-£ t J 

The resulting algorithm may be described as follows. Compute House- 
holder transformations to triangularize C t and also apply these transfor¬ 
mations to C % and d: 

(21.11) C,IC,:C 2 :d] = 

Compute the m 2 x m, matrix £ , as the solution of the triangular system 

Et£i = Ei 


( 21 . 12 ) 

Compute 

(21.13) 


E 2 — E 2 — £ t £ 2 
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and 


(21.14) 




Compute Householder transformations to triangularize £ t and also apply 
these transformations to /: 


(21.15) 


&&:/! = 


% 

0 

0 


9 

0 


}« — M, 

}m, + m. 


— n — 1 


Finally compute the solution vector 



by solving the triangular system 


(21.16) 



This algorithm requires no two-dimensional arrays of computer storage 
other than those required to store the initial data since quantities written 
with a tilde can overwrite the corresponding quantities written without a tilde 
and quantities written with a circumflex can overwrite the corresponding 
quantities written with a tilde. 

The coding of this algorithm can also be kept remarkably compact, as is 
illustrated by the ALGOL procedure decompose given in Bj&rck and Golub 


(1967). Suppose the data matrix 
=> m, + m 2 and the data vector 



is stored in an m x n array, A, with m 
is stored in an m-array, b. Steps (21.11) 


and (21.15) can be accomplished by the same code, essentially Steps 2-12 of 
Algorithm HFTI (14.9). When executing Eq. (21.11) arithmetic operations are 
performed only on the first m, rows of [A : b] but any interchanges of columns 
that are required must be done on the full m-dimensional columns of A. 

Steps (21.12) to (21.14), which may be interpreted as Gaussian elimina¬ 
tion, are accomplished by the operations 


(21.17) 

*m := ^ 
a lt 



(21.18) 

a tj- 

a tJ :=- 

% a ik o kJ 

• ■ 

II 

w 

■ 

• 

• 

2 

i-* 

(21.19) 

a t j != fly 

a, k a kJ 


(21.20) 

b{ != b ( 

Ji a,A 



i:=m, + 1, 

...,m 
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As a numerical example of this procedure, let us again consider Problem 
LSE with the data given in Eq. (20.25) to (20.28). For these data the matrix 
g, of Eq. (21.11) can be taken to be the 1 x 1 identity matrix. Using Eq. 
(21.11) to (21.16) we compute 



LINEAR LEAST SQUARES WITH 
LINEAR EQUALITY CONSTRAINTS 
BY WEIGHTING 


An observation that many statisticians and engineers have made is the 
following. Suppose one has a linear least squares problem where one would 
like some of the equations to be exactly satisfied. This can be accomplished 
approximately by weighting these constraining equations heavily and solving 
the resulting least squares system. Equivalently, those equations which one 
does not necessarily want to be exactly satisfied can be downweighted and 
the resulting least squares system can then be solved. 

In this chapter we shall analyze this formal computational procedure for 
solving Problem LSE (20.1). Basically the idea is simple. Compute the solution 
of the least squares problem: 

( 22 . 1 ) 


(using Householder's method, for example) for a "small" but nonzero value of e. 
Then the solution x(e) of the least squares problem (22.1) is "close" [in a sense 
made precise by inequality (22.38) and Eq. (22.37)] to the solution X of Problem 
LSE. 

This general approach is of practical interest since some existing linear 
least squares subroutines or programs can effectively solve Problem LSE by 
means of solving the system of Eq. (22.1). 

An apparent practical drawback of this idea is the fact that the matrix of 
problem (22.1) becomes arbitrarily poorly conditioned [assuming Rank ( C) 
< n] as the “downweighting parameter” e is made smaller. This observation 
certainly does limit the practicality of the idea if problem (22.1) is solved by 
the method of normal equations [see Eq. (19.1)]. Thus the normal equation 
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for problem (22.1) is 

(CPC + ei&Eyx ~C T d+ cWf 

and unless the matrices involved have very special structure, the computer 
r e pre sen tation of the matrix CPC + e i E T E will be indistinguishable from 
the matrix CPC (and CPd + e^EPf will be indistinguishable from (Pd) when 
f is sufficiently small. 

Nevertheless, Powell and Reid (1968a and 1968b) have found experi¬ 
mentally that a satisfactory solution to a disparately weighted problem such 
as Eq. (22.1) can be computed by Householder transformations if care is 
taken to introduce appropriate row interchanges. The reader is referred to 
Chapter 17, in which the principal theoretical results of Powell and Reid 
(1968a) are presented in the form of three theorems. 

In Powell and Reid (1968a) the interchange rules preceding the-construo- 
tion of the kth Householder transformation are as follows. First, do the usual 
column interchange as in Algorithm HFTI (14.9), moving the column whose 
euclidean length from row k through m is greatest into column k. Next scan 
the elements in column k from row k through m. If the element of largest 
magnitude is in row /, interchange rows / and k. 

This row interchange rule would generally be somewhat overcautious for 
use in problem (22.1). The main point underlying the analysis of Powell and 
Reid (1968a) is that a very damaging loss of numerical information occurs if 
the pivot element v, [using the notation of Eq. (10.5) to (10.11)] is significant 
for the problem but of insignificant magnitude relative to some dements v„ 
i > p. By “significant for the problem” we mean that the solution would be 
significantly changed if r, were replaced by zero. But if \v,\ is sufficiently 
small relative to some of the numbers |v,|, i > p, then v, will be computa¬ 
tionally indistinguishable from zero in the calculation of s and u, by Eq. (10.5) 
and (10.7), respectively. 

Suppose the nonzero elements of C and E in problem (22.1) are of the 
same order of magnitude so that any large disparity in sizes of nonzero ele¬ 
ments of the coefficient matrix of Eq. (22.1) is due to the small multiplier e. 
Then the disastrous loss of numerical information mentioned above would 
generally occur only if the pivot element v, [of Eq. (10.5)] were chosen from 
a row of e£ (say, e, — ee„) while some row of C (say, row /) has not yet been 
used as a pivot row and contains an element (say, c, f ) such that |c w |>e|e /y |. 
This situation can be avoided by keeping the rows in the order indicated in 
Eq. (22.1) so that all rows of C are used as pivot rows before any row of eE 
is used. Here we have used the symbols eE and C to denote those matrices or 
their successors in the algorithm. 

We now turn to an analysis of convergence of the solution vector of 
problem (22.1) to the solution vector of Problem LSEase —* 0. First consider 
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the special case of Problem LSE in which the matrix C is diagonal and E is 
an identity matrix. This case is easily understood, and it will be shown 
subsequently that the analysis of more general cases is reducible to this special 
case. 

Define 


( 22 . 2 ) 



and 



0 0 • • • 


o • • • 



Si ^ S mt > 0 


(22.3) 


£ = /. 


Together the following two lemmas show that the solution of the down¬ 
weighted least squares problem (22.1) converges to the solution of Problem 
LSE, as f —*■0, for the special case of C and E as given in Eq. (22.2) and 

(22.3) . 

(22.4) Lemma 

Suppose that C and E are given in Eq. (22.2) and (22.3). Then the solu¬ 
tion of Problem LSE using the notation of Eq. (20.2) and (20.3) is 
given by 


(22.5) 



i —■ 1,.. •, m j 
/ = m, + 1,... ,n 


The length of EX— f is 


( 22 . 6 ) 





1/2 


(22.7) Lemma 

Let C and E be as in Eq. (22.2) and (22 J). The unique least squares 
solution of problem (22.1) is given by 


2 -n-i 


( 22 . 8 ) = m = 


f+'*('--4) K‘+6)1 


ft i = m, -f I,..., n 


i—l,... t m 


Lemma (22.7) can be easily verified by forming and solving the normal 
equations (C^C + e 1 E T E)x = (C?d -f e*E T /). With the hypotheses present 
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on C and E this system has a diagonal coefficient matrix from which Eq. (22.8) 
easily follows. 

For convenience in expressing the difference between 5(e) and 5 define 
the vector-valued function A(e) by 

(22.9) 5Xe) -*, = e*AXf) / = 1. n 


Thus, using Eq. (22.S) and (22.8), 


( 22 . 10 ) 


and 




. 

,0 / = JW| + 1,. *., n 


m 


s t 


-2 


(22.il) m)\ 


f,-± 




i— l,...,m, 


lO i = m t + 


The residual vector for problem (22.1) is 
<“‘ 2) (,/) - [,*]*'> - 0 - [.«> + “ m 


“L- 


—e*CA(e) 

£5) - e»£A(e) 


) 


Introduce the weighted euclidean norm 


m, terms n terms 

(22.13) II-IL — £(•>+ ”•+(•)* + WO* + • • • + WO*] 1 ' 1 


then 

(ai4) Ktf)-O f) [-‘ ,|ic ‘ ( ' )i1 ' 

+ ||/-£5 + e*£A(0|l 2 


From Eq. (22.10) we see that 



as e 


0 


while from Eq. (22.14) we see that 

P-Q* ) IH l/ -“ 11 m '-° 
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For practical computation one is interested in the question of how small 
e must be to assure that x(e) and St are indistinguishable when their compo¬ 
nents are represented as computer words with a relative precision of if. If all 
components of the vector St are nonzero, this condition is achieved by requir¬ 
ing that 

(22.15) l«We)l£*l*l *=1.» 

This will be satisfied for all |e| <, e 0> where 

(22.16) el = min, { |}fl^|| - 4lh * <>} 

Using Eq. (22.9) and (22.11) the vector norm of the difference between 
5(e) and St is bounded as follows: 

(22.17) \\m - All ^ 

5 mi 

Thus in order to have || 5(e) — St || ^ if || St || it suffices to choose |e| <; e 0 , 
where 

(22.18) el = 

Note that Eq. (22.18) does not apply if ||/— 5|| = 0. In this case, however, 
Eq. (22.1) is consistent and has the same solution for all e & 0. 

We now consider Problem LSE for the more general case in which C is 
m t x nofrankifc, <; m, < n,Eism t x tt, and the matrix [C r : E*] is of rank 
it. It is further assumed that the constraint equations Cx = d are consistent. 

By appropriate changes of variables we. shall reduce this problem to the 
special one previously considered. First note that for | e | < 1 the least squares 
problem (22.1) is exactly equivalent to the least squares problem 

(1 — e*) 1/a C"| r(i -e*) I/l «ri 

. cv _i L </’ J 

'-ra - '-\a 

Next we introduce a uniform rescaling of problem (22.19), multiplying all 
coefficients and right-side components by (1 — e*) _l/1 to obtain the problem 


(22.19) 
where 

( 22 . 20 ) 


( 22 . 21 ) 
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where 


p = e(l- 


Introduce a QR decomposition of E ': 


( 22 . 22 ) 




where Q is (m, + m a ) x (i», + m 2 ) and orthogonal and R is n x n and 
nonsingular. With 

(22.23) g - Qf, Rx = y 

we have the equivalent least squares problem 



«,{ 

CR“>' 


*<# T 

(22.24) 

»{ 

ph 

ys 

/>* 


Mi+m,- n{ 

0 



m, 

m, + m x 


Let 


(22.25) CR l = t/SK r 

be a singular value decomposition of CR -1 [see Theorem (4.1)]. Since C is 
of rank k„ CR~ l is also, and we have 



where a, ^ ^ a*, > 0. 

Partition g into two segments, 


* - n , 

+ m 1 ~n 

Then with Vy — z the least squares problem of Eq. (22.24) is equivalent 
to the least squares problem 


S' 


' U T d ' 

Pi. 



.0. 


. Pf* . 


(22.26) 
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Since Cx — d is consistent, 


(22.27) 



Thus problem (22.1) has been transformed to problem (22.26), where the 
coefficient matrix consists of a zero matrix and two diagonal matrices, one of 
which is a scalar multiple of the identity matrix. 

From Lemmas (22.4) and (22.7) it follows that as p tends to zero the solu¬ 
tion 2(p) of problem (22.26) converges to the (unique) solution 2 of the prob¬ 
lem 

(22.28) minimize ||; - y r g, ||* + || gi ||? 

subject to Sz — U T d 

On the other hand, by means of the substitutions 

(22.29) Rx = y 
and 

(22.30) V T y = z 

where R and V are defined by Eq. (22.22) and (22.25), respectively. Problem 
LSE (20.1) is converted to problem (22.28). 

Thus using Eq. (22.29) and (22.30) we obtain 

(22.31) x(p) = R~'V2(p) 

as the (unique) solution of problem (22.1) with p = e(l — e 2 )" I/2 and 

(22.32) X = R-'V2 

as the (unique) solution of Problem LSE (20.1). 

To express the difference between x(p) and £, note that 2(p) and f satisfy 


(22.33) 


2(p) - f = p 2 h(p) 
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with the vector-valued function h given by Eq. (22.10) with appropriate 
changes of notation, and thus 

(22.34) x(p) - i = p 1 R~ i Vh(j>) 

In order to apply the bounds in Eq. (22.11) using the quantities appearing 
in problem (22.26), define 

(22.35) et = U r d 
and 

(22.36) g' = V r gl 
Then, using Eq. (22.11) and (22.34), 

\\m - *ii <; p*\\r- 1 ik;*[|J(gj - ffj 1 

where g\ and d\ are the fth components of the vectors of Eq. (22.35) and 

(22.36) , respectively. 

For i 0, one obtains the relative precision ||x(p) — *|| <, 9 ||*|| by 
choosing 0 <p<Lp 9 where 

pi =vn*n4.(iiii-ii[U(*{ -f)7 /i r 

Then defining 

(22.37) f 0 == 

the solution x(e) of Eq. (22.1) satisfies 

(22.38) \\m- *11^ n II *11 
for f satisfying 0 < e e 0 . 

It should be emphasized that the artifact of considering the various 
extended problems (22.19), (22.21), (22.24), (22.26), and (22.28) and the asso¬ 
ciated coordinate changes is only for the purpose of proving that the solution 
of problem (22.1) converges to the solution of Problem LSE (20.1) as e tends 
to zero. One should not lose sight of the fact that in any actual computational 
procedure based on these ideas, one can simply solve problem (22.1) directly, 
say, by Householder transformations. 

Although the value of e 0 given by Eq. (22.37) is an upper bound for | e | in 
order to achieve full relative precision if, it involves quantities that one would 
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not generally compute in solving problem (22.1). Note that there is no positive 
lower bound on permissible values of | e | imposed by the mathematical analysis 
of the problem or the numerical stability of the Householder method as 
analyzed in Powell and Reid (1968a). The only practical lower limit is set by 
the computer’s floating point underflow value, L, referred to in Chapter IS. 

For example, consider a machine with ti = 10"* and Z, = 10~*'. Suppose 
all nonzero data in the matrices C and E and the vectors d and/ are approxi¬ 
mately of unit magnitude. 

In practice one would probably need to have e < if Ul — 10~ 4 and e > L 
= 10~**. With this wide range available one might, for example, decide to 
set e = 10" 1 *. 

As a numerical illustration of this weighted approach to solving Problem 
LSE, consider the data given in Eq. (20.25) to (20.28). We now formulate this 
as the weighted least squares problem 



0.4087 

0.1593 ‘ 

Tv "1 

0.1376 ' 

(22.39) 

0.4302e 

0.3516e 


0.6593e 


0.6246e 

0.3384e 


_0.9666e 


Problem (22.39) was solved on a UNIVAC 1108 computer using sub¬ 
routine HFTI (Appendix C) with mixed precision (precisions: if = 2" M == 

10 *- > and <a = 2” = 10-«”) for 40 values of e (e -10-% r = 1.40). 

Let x M denote the solution obtained when e — 10 _r . 


Table 22.1 


r 

&<r) 

1 

3.7 X 10-a 

2 

3.7 x 10-« 

3 

3.6 x 10-‘ 

4 

6.3 X I0-» 

5 

2.6 X 10-’ 

6 

9.1 x 10-* 

7 

9.1 x 10-« 

8 

1.2 X 10-’ 

9 

4.5 x 10-* 

10 

• 

5.8 X 10-« 

• 

• 

• 

36 

• 

• 

8.7 x 10-« 

37 

3.6 x 10-» 

38 ) 

40 J 

e too small. Numbers multiplied by c 
underflow to zero. 
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A “true” solution vector x was computed by the method of Chapter 20 
using full double precision (10~") arithmetic. To 12 figures this vector was 



—1.17749898217' 

3.88476983058. 


The relative error of each solution x ,r> was computed as 



A selection of values of 6 W is given in Table 22.1. 

From Table 22.1 we see that any value of e from 10~ 4 to 10~ 17 gave 
reasonable single precision (q == 10 _l ') agreement with the “true” answer, St. 


EXERCISE 


(22.40) In Eq. (22.1) the matrix C is m t x n, E is m x x n, and m — m, 
+ m a . Let H, denote the first Householder matrix that would be 

constructed when triangularizing [£} Define 



Show that S — lim exists and give formulas for computing 

#-•0 

vectors u t and u a of dimensions mi and m 2 , respectively, and a scalar 
b (as functions of the first column of ^such that 

5 = + 


Show that the product matrix has zeros below the first ele¬ 
ment in the first column and thus that an appropriately constructed 
sequence of m x matrices like R can be used to zero all elements 

below the diagonal in the first m, columns of j^J. Show that this 
sequence of operations has the same effect as Eq. (21.11) to (21.14). 
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Section 1. INTRODUCTION 

There are many applications in applied mathematics, physics, statistics, 
mathematical programming, economics, control theory, social science, and 
other fields where the usual least squares problem must be reformulated by 
the introduction of certain inequality constraints. These constraints constitute 
additional information about a problem. 

We shall concern ourselves with linear inequality constraints only. A vari¬ 
ety of methods have been presented in the literature. We mention particu¬ 
larly papers that have given serious attention to the numerical stability of 
their methods: Golub and Saunders (1970), Gill and Murray (1970), and 
Stoer (1971). 

The ability to consider least squares problems with linear inequality con¬ 
straints allows us, in particular, to have such constraints on the solution as 
nonnegativity, or that each variable is to have independent upper and lower 
bounds, or that the sum of all the variables cannot exceed a specified value 
or that a fitted curve is to be monotone or convex. 

Let fbean m a x n matrix, / an m*-vector, G an m x n matrix, and h 
an m-vector. The least squares problem with linear inequality constraints will 
be stated as follows: 

(23.1) Problem LSI 

Minimize || Ex — f || subject to Gx ^ h. 
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The following important special cases of Problem LSI will also be treated in 
detail: 

(23.2) Problem NNLS ( Nonnegative Least Squares) 

Minimize || Ex — f j| subject to x ^ 0. 

(23.3) Problem LDP ( Least Distance Programming) 

Minimize |j x|| subject to Gx > h. 

Conditions characterizing a solution for Problem LSI are the subject of 
the Kuhn-Tucker theorem. This theorem is stated and discussed in Section 
2 of this chapter. 

In Section 3 Problem NNLS is treated. A solution algorithm, also called 
NNLS, is presented. This algorithm is fundamental for the subsequent 
algorithms to be discussed in this chapter. A Fortran implementation of 
Algorithm NNLS is given in Appendix C as subroutine NNLS. 

In Section 4 it is shown that the availability of an algorithm for Problem 
NNLS makes possible an elegantly simple algorithm for Problem LDP. 
Besides stating Algorithm LDP in Section 4, a Fortran implementation, sub¬ 
routine LDP, is given in Appendix C. 

The problem of determining whether or not a set of linear inequalities 
<?jc ;> h is consistent and if consistent finding some feasible vector arises in 
various contexts. Algorithm LDP can of course be used for this purpose. 
The fact that no assumptions need be made regarding the rank of G or the 
relative row and column dimensions of G may make this method particularly 
useful for some feasibility problems. 

In Section S the general problem LSI, having full column rank, is treated 
by transforming it to Problem LDP. Problem LSI with equality constraints 
is treated in Section 6. 

Finally in Section 7 a numerical example of curve fitting with inequality 
constraints is presented as an application of these methods for handling 
constrained least squares problems. A Fortran program, PROG6, which 
carries out this example, is given in Appendix C. 

Section 2. CHARACTERIZATION OF A SOLUTION 

The following theorem characterizes the solution vector for Problem LSI: 

(23.4) Theorem ( Kuhn-Tucker Conditions for Problem LSI) 

An n- vector His a solution for Problem LSI (23.1) if and only if there 
exists an m-vector 9 and a partitioning of the integers l through m 
into subsets S and § such that 


(23.5) 


G r 9 = E T (Ek - /) 
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(23.6) />, = 0for/ e 8, f,>0 for i e g 

(23.7) % S: 0 for i s 8, j>, = 0 for / e g 

where 

(23.8) t — GX — h 

This theorem may be interpreted as follows. Let gj denote the ith row 
vector of the matrix G. The Ah constraint, gfx ^ h„ defines a feasible half- 
space, {x:gjx ^ h,}. The vector g, is orthogonal (normal) to the bounding 
hyperplane of this halfspace and is directed into the feasible halfspace. The 
point X is interior to the halfspaces indexed in $ (g for slack ) and on the 
boundary of the halfspaces indexed in 8 (8 for equality). 

The vector 

p = E T (EX - /) 

is the gradient vector of q(x) = ^j| Ex — /|| 2 at x = X. Since = 0 for 
i^S Eq. (23.5) can be written as 

(23.9) £ U-g>) = ~P 

which states that the negative gradient vector of p at X is expressible as a non* 
negative {$, ^ 0) linear combination of outward-pointing normals (— g,) to 
the constraint hyperplanes on which X lies (i e 8). Geometrically this means 
that the negative gradient vector —p lies in the convex cone based at the point 
X and generated by the outward-pointing normals —g,. 

Any perturbation u of X such that X + u remains feasible must satisfy 
u r g, ;> 0 for all i e 8. Multiplying both sides of Eq. (23.9) by such a vector 
u r and using the fact that the ^ 0, it follows that u also satisfies u T p ^ 0. 
From the identity p(X + «) = p(X) +• u T p + ||£b|| a /2, it follows that no 
feasible perturbation of X can reduce the value of p. 

The vector (or the negative of this vector) which occurs in the Kuhn- 
Tucker theorem is sometimes called the dual vector for the problem. Further 
discussion of this theorem, including its proof, will be found in the literature 
on constrained optimization [see, e.g., Fiacco and McCormick (1968), p. 256]. 


Section 3. PROBLEM NNLS 

Problem NNLS is defined by statement (23.2). We shall state Algorithm 
NNLS for solving Problem NNLS. The finite convergence of this algorithm 
will be proved. 

We are initially given the m t x n matrix E, the integers m t and n, and 
the Ma-vector /. The n-vectors w and z provide working space. Index sets 9 
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and Z wiO be defined and modified in the course of execution of the algorithm. 
Variables indexed in the set Z will be held at the value zero. Variables indexed 
in the set <P will be free to take values different from zero. If such a variable 
takes a nonpositive value, the algorithm will either move the variable to 
a positive value or else set the variable to zero and move its index from set <P 
to set Z. 

On termination x will be the solution vector and w will be the dual vector. 

(23.10) Algorithm NNLS{£, m lt n,f, x, w, x, <P, Z) 

Step Description 

1 Set <P := null, Z := {1,2,...,«}, and * := 0. 

2 Compute the n-vector w := ETif — Ex). 

3 If the set Z is empty or if w,£ 0 for all j e Z, go to 
Step 12. 

4 Find an index / e Z such that w, = max {tv,: j e Z}. 

5 Move the index t from set Z to set (P. 

6 Let Ef denote the m t x n matrix defined by 

_ (column j of E if / e <P 

Column y of E,:=) 0 jf , g z 

Compute the /i-vector z as a solution of the least squares 
problem EfZ =f Note that only the components z„j e <9, 
are determined by this problem. Define z, := 0 fox j e Z. 

7 If z, > 0 for all j € <P, set x :=* z and go to Step 2. 

8 Find an index q e 9 such that xj(x t — z f ) = min 

[xjKxj - Zj): z, <. 0 ,j e <P}. 

9 Set a xj(x, - z t ). 

10 Set x :== x + — *)• 

11 Move from set (P to set Z all indices j e (P for which x } — 
0. Go to Step 6. 

12 Comment: The computation is completed. 

On termination the solution vector x satisfies 

(23.11) x, > 0 j g <P 

(23.12) x, = 0 j € Z 

and is a solution vector for the least squares problem 

(23.13) EfX^f 
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The dual vector w satisfies 

(23.14) 

w t = 0 j e <P 

(23.15) 

w,<. 0 j g 2 

and 


(23.16) 

w = E T (f - Ex) 


Equations (23.11), (23.12), (23.14), (23.15), and (23.16) constitute the 
Kuhn-Tucker conditions [see Theorem (23.4)] characterizing a solution vec¬ 
tor x for Problem NNLS. Equation (23.13) is a consequence of Eq. (23.12), 
(23.14), and (23.16). 

Before discussing the convergence of Algorithm NNLS it will be conven¬ 
ient to establish the following lemma: 


(23.17) Lemma 


(23.18) 


Let Abeanm X n matrix of rank n and let b be an m-vector satisfy¬ 
ing 



(23.19) 


co > 0 


If % is the least squares solution of Ax s b, then 


(23.20) *.>0 

where St m denotes the nth component of St. 


Proof: Let Q be an m x m orthogonal matrix that zeros the sub¬ 
diagonal elements in the first n — 1 columns of A, thus 


(23.21) 


Q[A:b] = 

W W-* 

n 1 



1 

vj)m — n + 1 
1 


where R is upper triangular and nonsingular. 

Since Q is orthogonal the conditions (23.18) imply 


( 23 . 22 ) 


R T u = 0 
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and 

(23.23) s T u -f- fv — to > 0 

Since R is nonsingular, Eq. (23.22) implies that u — 0. Thus Eq. (23.23) 
reduces to 

(23.24) t T v = o) > 0 

From Eq. (23.21) it follows that the nth component X, of the solution 
vector A is the least squares solution of the reduced problem 

(23.25) tx, =s « 

Since the pseudoinverse of the column vector t is t T ftt T t), the solution of 
problem (23.25) can be immediately written as 

(23.26) *. = £7 = % >° 

which completes the proof of Lemma (23.17). 

Algorithm NNLS may be regarded as consisting of a main loop. Loop A, 
and an inner loop. Loop B. Loop B consists of Steps 6-11 and has a single 
entry point at Step 6 and a single exit point at Step 7. 

Loop A consists of Steps 2-5 and Loop B. Loop A begins at Step 2 and 
exits from Step 3. 

At Step 2 of Loop A the set 9 identifies the components of the current 
vector x that are positive. The components of x indexed in Z are zero at this 
point. 

In Loop A the index t selected at Step 4 selects a coefficient not presently 
in set <P that will be positive (by Lemma (23.17)] if introduced into the solu¬ 
tion. This coefficient is brought into the tentative solution vector z at Step 6 
in Loop B. If all other components of z indexed in set (P remain positive, then 
at Step 7 the algorithm sets x := z and returns to the beginning of Loop A. 
In this process set 9 is augmented and set 1 is diminished by the transfer of 
the index t. 

In many examples this sequence of events simply repeats with the addition 
of one more positive coefficient on each iteration of Loop A until the termina¬ 
tion test at Step 3 is eventually satisfied. 

However, if some coefficient indexed in set 9 becomes zero or negative in 
the vector z at Step 6 , then Step 7 causes the algorithm to remain in Loop B 
performing a move that replaces x by x -j- «(z — x), 0 < a <, 1 , where a 
is chosen as large as possible subject to keeping the new x nonnegative. Loop 
B is repeated until it eventually exits successfully at Step 7. 
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The finiteness of Loop B can be proved by showing that all operations 
within Loop B are well defined, that at least one more index, the index called 
q at that point, is removed from set <P each time Step 11 is executed, and 
that z, is always positive [Lemma (23.17) applies here]. Thus exit from Loop 
B at Step 7 must occur after not more than it — 1 iterations within Loop B, 
where it denotes the number of indices in set 9 when Loop B was entered. 
In practice Loop B usually exits immediately on reaching Step 7 and does not 
reach Steps 8—11 at all. 

Finiteness of Loop A can be proved by showing that the residual norm 
function 

Pix) = 11/ — Ex\\ 

has a strictly smaller value each time Step 2 is reached and thus that at Step 
2 the vector x and its associated set 9 = {<: x, > 0} are distinct from all 

previous instances of x and 9 at Step 2. Since (Pisa subset of the set {1,2. 

«} and there are only a finite number of such subsets. Loop A must terminate 
after a finite number of iterations. In a set of small test cases it was observed 
that Loop A typically required about iterations. 

Updating the QR Decomposition of E 

The least squares problem being solved at Step 6 differs from the problem 
previously solved at Step 6 either due to the addition of one more column of E 
into the problem at Step 5 or the deletion of one or more columns of E at 
Step 11. Updating techniques can be used to compute the QR decomposition 
for the new problem based upon retaining the QR decomposition of the 
previous problem. Three updating methods are described in Chapter 24. 
The third of these methods has been used in the Fortran subroutine NNLS 
(Appendix C). 

Coping with Finite Precision Arithmetic 

When Step 6 is executed immediately after Step 3 the component z, com¬ 
puted during Step 6 will theoretically be positive. If z, is not positive, as can 
happen due to round-off error, the algorithm may attempt to divide by zero 
at Step 8 or may incorrectly compute a = 0 at Step 9. 

This can be avoided by testing z, following Step 6 whenever Step 6 has 
been entered directly from Step 5. If z, <. 0 at this point, it can be interpreted 
to mean that the number w, computed at Step 2 and tested at Steps 3 and 4 
should be taken to be zero rather than positive. Thus one can set w, := 0 and 
loop back to Step 2. This will result either in termination at Step 3 or the 
assignment of a new value to t at Step 4. 

At Step 11 any x, whose computed value is negative (which can only be 
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due to round-off error) should be treated as being zero by moving its index 
from set <P to set Z. 

Hie sign tests on z„ i e 9, at Steps 7 and 8 do not appear to be critical. 
The consequences of a possible misclassification here do not seem to be 

A Fortran subroutine NNLS implementing Algorithm NNLS and using 
these ideas for enhancing the numerical reliability appears in Appendix C. 

8 ootlon 4. PROBLEM LOP 

The solution vector for Problem LDP (23.3) can be obtained by an 
appropriate normalization of the residual vector in a related Problem NNLS 
(23.2). This method of solving Problem LDP and its verification was brought 
to the authors* attention by Alan Cline. 

We are given the m x n matrix G, the integers m and n, and the m-vector 
h. If the inequalities Gx^hare compatible, then the algorithm will set the 
logical variable p — true and compute the vector X of minimal norm satisfy¬ 
ing these inequalities. If the inequalities are incompatible, the algorithm will 
set f — falsb and no value will be assigned to X. Arrays of working space 
needed by this algorithm are not explicitly indicated in the parameter list. 

(23.27) Algorithm LDP(G, m, n, h, X, p) 

Step Description 

1 Define the (d + 1)xm matrix E and the (n + l)-vector / 

n 

as E := jjp J and /:= [0,..., 0, l] r . Use Algorithm 

NNLS to compute an m -vector d solving Problem NNLS: 
Minimize \\Eu — f\\ subject to n.;> 0. 

2 Compute the (« + l)-vector r := M —f. 

3 If Hr || = 0, set p := false and go to Step 6. 

4 Set p := true. 

5 For j := 1,...,», compute X, := —rj/r m+1 . 

6 The computation is completed. 

Proof of Validity of Algorithm LDP 

First consider the Problem NNLS, which is solved in Step 1 of Algorithm 
LDP. The gradient vector for the objective function $|| Eu — f || 2 at the solu¬ 
tion point d is 


( 23 . 28 ) 


p — E T r 
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From the Kuhnr-Tucker conditions (Theorem (23.4)] for this Problem NNLS 
there exist disjoint index sets 6 and S such that 

(23.29) £ u $ = {1,2,.... m] 

(23.30) a, = 0 for i e S, u, > 0 for i e S 

and 


(23.31) p, ;> 0 for i e 6, p, = 0 for i e g 

Using Eq. (23.28) to (23.31) we obtain 


(23.32) 

IMP = r r r = r'lEH — f] 


= P r “ ~ 1 — i 

Consider the case in which ||/-j| > 0 at Step 3. From Eq. (23.32) this 
implies that /•„+, < 0, so division by r„+ t at Step 3 is valid. Using Eq. (23.31) 
and (23.32) and the equations of Steps 2 and 3, we establish the feasibility 
of X as follows: 


O^p — E T r 

(23.33) 


Therefore, 

(1 

% 

1 

*5 

(23.34) 

GX^h 


From Eq. (23.31) and (23.33) it follows that the rows of the system of 
inequalities of Eq. (23.34) indexed in set § are satisfied with equality. The 
gradient vector for the objective function x || 2 of Problem LDP is simply x. 
The Kuhn-Tucker conditions for X to minimize £j|x|p subject to Gx^h 
require that the gradient vector X must be representable as a nonnegative 
linear combination of the rows of G that are associated with equality condi¬ 
tions in Eq. (23.34), i.e., the rows of G indexed in set S- 
From Steps 2 and 3 and Eq. (23.32) we have 
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Noting the sign conditions on <2 given in Eq. (23.30) completes the proof that 
X is a solution of Problem LDP. 

It is clearly the unique solution vector since, if x is a different solution 
vector, then ||Jf|| = ||i|| and the vector x = $(x + X) would be a feasible 
vector having a strictly smaller norm than X, which contradicts the fact that X 
is a feasible vector of minimum norm. 

Now consider the case of ||r|| = 0 at Step 3. We must show that the 
inequalities Gx 2> h are inconsistent. Assume the contrary, i.e., that there 
exists a vector x satisfying Gx > h. Define 


Then 


q — Gx — h = [G : A] 





This last expression cannot be 2 ero, however, because q ^ 0 and « ^ 0 . 
From this contradiction we conclude that the condition ||r|| = 0 implies 
the inconsistency of the system Gx > h. This completes the mathematical 
verification of Algorithm LDP. 


Section 5. CONVERTING PROBLEM LSI 
TO PROBLEM LOP 


Consider Problem LSI (23.1) with the m 2 X n matrix E being of rank n. 
In various ways as described in Chapters 2 to 4 one can obtain an orthogonal 
decomposition of the matrix E: 


(23.35) 



where Q is m a X nt 2 orthogonal, K is n x n orthogonal, and R is n x n 
nonsingular. Furthermore, the matrix R can be obtained in triangular or 
diagonal form. 

Introduce the orthogonal change of variables 


(23.36) 


x = Ky 


The objective function to be minimized in Problem LSI can then be written as 
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«-» —-icKi-rar 

— Il/i — ^.yll* + ll/ill a 

where 

(23.38) /, = #/ i — \,2 
With a further change of variables, 

(23.39) z — Ry — f i 
we may write 

(23.40) **) = 11*11*+ 117,11* 

The original problem LSI of minimizing || / — Ex || subject to Gx ^ h is 
thus equivalent, except for the additive constant ||7,ll* in the objective 
function, to the following Problem LDP: 

(23.41) Minimize || z || 

subject to GKR~ l z > h - GKR~ l j l 

If a vector t is computed as a solution of this Problem LDP, then a solu¬ 
tion vector X for the original Problem LSI can be computed from Eq. (23.39) 
and (23.36). The squared residual vector norm for the original problem can 
be computed from Eq. (23.40). 


Section 6. PROBLEM L8I WITH EQUALITY 
CONSTRAINTS 


Consider Problem LSI (23.1) with the addition of a system of equality 
constraints, say C m ,. n x=d, with Rank(C)=mj < n and Rank ([C T :E r \)=n. 
The equality constraint equations can be eliminated initially with a corre¬ 
sponding reduction in the number of independent variables. Either the 
method of Chapter 20 or that of Chapter 21 is suitable for this purpose. 

Using the method of Chapter 20, introduce the orthogonal change of 
variables 


(23.42) 


Lv,J 


}« - mi 


where K triangularizes C from the right: 
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C' 


re. 

o • 

(23.43) 

£ 

K = 

E t 



.G. 


Pi 

g 2 . 


m, n — m x 


Then fi, is determined as the solution of the lower triangular system £,y x = d, 
and fi 2 Is the solution of the following Problem LSI: 

(23.44) Minimize || £& - (f - £j>,)|| 

subject to & 2 y 2 ^ h — & t y, 

After solving problem (23.44) for fi 2 the solution 5t can be computed using 
Eq. (23.42). 

If the method of Chapter 21 is used, one would compute Q U C U (? 2> 3, E u 
E 2 , and / using Eq. (21.11) to (21.14) and additionally solve for the matrix 
<?, in the upper triangular system 

(23.45) GtC, = G, 

Then is the solution of the following Problem LSI: 

(23.46) Minimize \\£tx t — /}[ 

subject to (G, — &i£ 2 )x 2 ;> h — G t 3 

and would be computed by solving the upper triangular system 

(23.47) £ lXi = i - C 2 X 2 

Section 7. AN EXAMPLE OF CONSTRAINED 
CURVE FITTING 

As an example illustrating a number of the techniques that have been 
described in this chapter we consider a problem of fitting a straight line to 
a set of data points where the line must satisfy certain constraints. PROG6, 
a Fortran main program that performs the computation for this example, is 
given in Appendix C. 

Let the data be given as follows: 


t 

w 

0.25 

0.5 

0.50 

0.6 

0.50 

0.7 

0.80 

L2 
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We wish to find a line of the form 

(23.48) f(t) - x lt + x, 

which fits these data in a least squares sense subject to the constraints 


(23.49) fit) ^ 0 

(23.50) /(0) 22 0 

(23.51) /(1) <£ 1 

This problem can be written as Problem LSI: 


(23.52) 

where 


Minimize ||.&c — f\\ 
subject to Gx h 


'0.25 1' 
„ 0.50 1 

“ 0.50 1 
0.80 1. 

r i o 




-l 



We compute an orthogonal decomposition of the matrix E in order to 
convert the Problem LSI to a Problem LDP as described in Section 5 of this 
chapter. Either a QR or a singular value decomposition of E could be used. 
We shall illustrate the use of a singular value decomposition. 




2.255 0.0 ' 

0.0 0.346 


'—0.467 0.884' 

0.884 -0.467. 
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Introduce the change of variables 

(23.53) x = SV T x - f t 

We then wish to solve the following Problem LDP: 

(23.54) Minimize || s || 

subject to Gx ^ h 

where 

-0.207 2.558' 

6 = GVS~ l = -0.392 -1.351 

0.599 -1.206. 

and 

'-1.300' 

It — h — Gf t = -0.084 

0.384. 

A graphical interpretation of this Problem LDP is given in Fig. 23.1. 
Each row of the augmented matrix [fi: 2] defines one boundary line of the 
feasible region. The solution point £ is the point of minimum euclidean norm 



Fit. 23.1 Graphical interpretation of the sample Problem LDP 
(23.54). 
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within the feasible region. This point, as computed by subroutine LDP, is 



Then using Eq. (23.53) we finally compute 

*- + /,) - P] 

The residual vector for the solution vector St is 


f = f ~ ESt = 


-0.034' 

-0.089 

0.011 

0.324. 



Fig. 23.2 Graph of solution line for the sample problem (23.48)- 
(23.51). 
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aad the residual norm is 

||f|| = 0.338 

The given data points (/,, w t ), i — 1,... ,4, and the fitted line, /(/) — 
0.621/ + 0.379, are shown in Fig. 23.2. Note that the third constraint, /(l) 
<; 1, is active in limiting how well the fitted line approximates the data points. 

The numerical values shown in describing this example were computed 
using a UNTVAC 1108 computer. Executing the same Fortran code on 
an IBM360/67 resulted in opposite signs in the intermediate quantities V ,/„ 
</, and f. This is a consequence of the fact that the signs of columns of the 
matrix V in a singular value decomposition are not uniquely determined. 
The difference in the number of iterations required to compute the singular 
value decomposition of the matrix E on the two computers having different 
word lengths resulted in a different assignment of signs in the matrices V 
and V. 


EXERCISES 

(23.53) Prove that if a Problem LSE has a unique solution without inequality 
constraints, then it has a unique solution with inequality constraints. 

(23.56) Show that the problem of minimizing a quadratic function fix) 
— \x T Bx + a T x, for positive definite B can be transformed to the 
problem of minimizing £ 11 w | |* by letting w = Fx — g for an appro¬ 
priate choice of the nonsingular matrix F and vector g. 

(23.57) If the function / of Ex. (23.56) is to be minimized subject to the 
constraints Cx = d and Gx ^ h, what are the corresponding con¬ 
straints for the problem of minimizing i||H'||*? 



M MODIFYING A QR DECOMPOSITION TO 

ADD OR REMOVE COLUMN VECTORS 


The key to a successful algorithm for the constrained Problem LSI of 
Chapter 23 is the ability to compute solutions to a sequence of least squares 
problems in an efficient manner. In Algorithm NNLS (23.10) it can be seen 
that the coefficient matrix has the property that one linearly independent 
column vector is added or removed at each step. 

Thus we discuss the following problem: Let n > 0. Let . .a. be a 

set of m-vectors. Consider the matrix 

(24-1) A = [e„ ..., OltlmKk 

Introduce a QR decomposition for A k : 

(24.2) QA k = 

Here R k iskx k upper triangular and nonsingular, while Q is orthogonal. 
Once Q and R k have been computed, the least squares solution of the system 

(24.3) A k x = b 
is given by 

(24.4) x = [Ri“: O]06 a Aib 

which requires very little additional work. 

We shall consider the problem of computing a QR decomposition for a 
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matrix obtained by deleting a column vector from A k or adjoining a new 
column vector to A k . We wish to take advantage of the prior availability of a 
QR decomposition of A k . 

We shall discuss three useful methods for updating the QR decomposi¬ 
tion. In view of Eq. (24.4) the updating of Q and R effectively provides an 
updating of At. 

Method 1 

We have Q stored as an explicit m x m orthogonal matrix and R stored 
as a k x k upper triangle. 

Adjoining a Vector 

Let a k+t be a vector linearly independent of the columns of A k . Form the 
augmented matrix 

(24.5) A kki — [A k ‘ a* +I ] 

Compute the product 

^*+1 = Q a k +1 

using Q of Eq. (24.2). Introduce a Householder matrix Q* + , so that the vector 

r k* I — Qk+ lh*+l 

has zero components in positions fc + 2,...,m. Then a QR decomposition 
of A k+i is given by 

e4 “'“[o] 

where 

Q = Qk+iQ 

and 

[MM 

Removing a Vector 

If the matrix A k of Eq. (24.1) is modified by the removal of the column 
vector a Jt forming the matrix 
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(24.6) An -1 — [fli,... > Bj-i, Bj+i,... ,0/,] 
we see that 

(24.7) Q^k~i — Ifu • • ■» fj- 1> P/+i» •••» r k ] 

where each r, is zero in entries i + 1.m. [See Golub and Saunders (1969) 

and Stoer (1971).] Define 

(24.8) Q - <?*-, ■ •• G J+t GjQ 

where the G, are Givens rotation matrices chosen so that the matrix 

(24.9) - [ J] 

is upper triangular. The matrix G, operates on rows i and i + 1 and produces 
a zero in position (/ + 1, /) of the product matrix ((?,••• G, QA t . t ). 

Method 1 requires m a computer storage locations for Q and /(/ + l)/2 
locations for R where / <£ m is the largest number of column vectors to be 
used simultaneously. 

Method 2 

Instead of storing Q explicitly as in Method 1 we now assume that Q is 
given as a product of k Householder transformations. 

(24.10) Q = Q k • •. Q t 
where each 

(24.11) Qt = l m + br'utuj 

is stored by retaining only the nonzero entries of the vectors u, plus one ad¬ 
ditional scalar associated with each vector. The matrix R is again stored as a 
k x k upper triangle. 

Adjoining a Vector 

If the set is enlarged to the linearly independent set [a if «*+,] 

compute the product b k+i — Q k ••• Qia k+l . Compute a Householder 
transformation , so that &+A+i = r*+i is zero in entries k + 2 t ...,m. 
It follows that 
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(24.12) <?*♦,(& • • • QftA k : a k+i ] = J*]: r* +I 

is a QR decomposition for the enlarged set. 

Removing a Vector 

Again define the matrix A k . t by Eq. (24.6). The data defining the QR 
decomposition of A k is of the form 



(24.13) Q k - QiA, 



j- 

1 k-j+1 

r 

Ru 

R?xt 

0- 

0 

R*i 

L v J 

. 0 

0 


}J- 1 

}k-j+l 
}m — k 


Denote the QR decomposition of A k . t by QA t _, — The matrix R will 

contain the same submatrix /?,, appearing in Eq. (24.13) as its leading 
(j — 1) x (j — 1) submatrix. Thus only the last k—j columns of R need 
to be determined. 

One approach is to replace 

RW 
R» 
o 

» « 

in storage by [a y+ i,..., a k ]. Then compute 
(24.14) Q t -1 • • • ...»<**] — S m x{k-n 

Next premultiply S by new Householder transformations, Qi, so that 

k-j 


&-> ffjS = 


R 

R 


la 


aa 


w-i 

}k~j 

}m — k + l 

with R 2i upper triangular. 

The QR decomposition of A k _, is then represented as 


L 0 


0*-i ffjQj -1 ••• QiA k -t — 


R\x Rn 

0 R u 
o o 
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Alternatively, the matrix S of Eq. (24.14) could be produced by the for 
mula 


(24.15) 


QjQ 


/+! 


• • « 


Qk-iQt 


R\i 

R'n 

0 




where R! lt and R! 2l are defined in Eq. (24.13). 

Method 2 requires (m + 1)/ storage locations to retain the data defining 
Q and R. Here / ^ m is the maximum number of columns being used simul¬ 
taneously. The version of Method 2 using Eq. (24.14) requires an additional 
m x n storage array to retain [a,,.... aJ so that any of these vectors will 
be available when needed in Eq. (24.14). 

The second version of Method 2, using Eq. (24.15), obviates the need 
to retain copies of the vectors a Jt a J+u ...,a k in storage. Thus all the data 
representing the QR decomposition of a matrix A k plus all the vectors a, not 
presently being regarded as columns of A k can be stored in one mxn storage 
array plus about 3m additional locations for bookkeeping purposes. 


Method 3 

In this method the entire contents of a storage array, W mx0l+li , which 
initially contains the data [A mxa : &„*,], is modified each time a new column 
vector is adjoined to or deleted from the basis. The column vectors of the 
upper triangular matrix R k of Eq. (24.2) will occupy some set of k columns of 
the array W. No record will be kept of the matrix Q of Eq. (24.2). 

Let <P k — { Pi, . p k ) be a subset of {1,2,..., n} identifying the 

columns of A that have been introduced into the basis. The order of the 
integers p, in the set <? k is significant. Let A k denote the m x k matrix consist¬ 
ing of those column vectors of A indexed in <P and ordered in the sequence 
Pi,Pi ,... ,p k . Let Q denote the orthogonal matrix satisfying Eq. (24.2). 
Then W will contain the m x (n 4- 1) matrix Q[A : b). This implies that the 
jth column vector of R k will be stored in column number Pj of W. In this 
method it may be found convenient to store explicit zeros for elements that 
are zeroed by Householder or Givens transformations. 

Adjoining a Vector 

Let p k+ 1 denote the index of a column of A to be adjoined to the basis. 
Form a new index set <?*+, consisting of p it ...,p k from (P* plus p k + 1 . Form 
the Householder transformation H that transforms column number p k+l of 
W to have zeros below row k + 1. Premultiply H times the entire array W. 


CHAP. 24 


MODIFYING A QR DECOMPOSITION 179 


Removing a Vector 

Assume again that (P* identifies the current basis. Let 1 <,j <. k and as¬ 
sume column number p, is to be removed from the basis. For i = j + l,j + 
2,... ,k form the Givens rotation matrix G t which operates on rows i — 1 
and i and zeros the (/, p, +i ) element of W. Premultiply G, times the entire 
array W. 

Form the new index set <P*_ t by setting p,:—p, for i — 1, 2,.. .— 1 
and setting p, := p, +i for / =y,y +1,..., k — 1. This method requires only 
the m x (n + 1) storage array W which initially contains the data [A : b]. If 
a copy of the initial data will be needed at later stages of the computation, 
then it must be saved in additional storage. 



JF PRACTICAL ANALYSIS OF 
LEAST SQUARES PROBLEMS 


Section 1. General Considerations 
Section 2. Left Multiplication of A and b by a Matrix G 
Section S. Right Multiplication of A by a Matrix H and Change of Variables 
x - H2 + ( 

Section 4. Append Additional Rows to [ A : b] 

Sections. Deleting Variables 
Section 6. Singular Value Analysis 


Section 1. GENERAL CONSIDERATIONS 

In this chapter we discuss the strategy of planning and interpreting the 
practical solution of a least squares problem. We consider the case of m n, 
as this is the central problem under consideration in this book. 

The person who originates a least squares problem must acquire data and 
develop or select a mathematical model and frequently a statistical model 
also. At some point in his work he reaches the stage at which he has a matrix 
A, and a vector b, and he desires to find a vector x that minimizes the 
euclidean norm of the residual vector 

(25.1) r = b - Ax 

Hie problem originator should also have information about the un¬ 
certainty of the data constituting A and b. Frequently he may have a priori 
information about the solution of his matrix problem. This may involve some 
knowledge as to what would constitute reasonable values for some or all of 
the solution components. It might also include information such as a require¬ 
ment that some or all solution components must be nonnegative. 

In very general terms there are two major considerations in the design of a 
computational approach to the problem and it is important to keep these 
points separate. 
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1. Computational error can be kept down to the point where it is negli¬ 
gible compared with uncertainty in the solution caused by uncertainty in the 
initial data. 

2. The combined effect of the a priori uncertainty in A and b and the con¬ 
dition number of A may produce a situation in which there are many signif¬ 
icantly different vectors x that reduce the norm of r to an acceptably small 
value. We shall discuss techniques for detecting this situation and some 
methods for controlling the selection of a particular solution from this set 
of candidate solutions. 

Expanding on point 1, we assume that the information regarding the un¬ 
certainty in A and b can be expressed by the statement that there are known 
numbers p and p such that any matrix of the form A -f- E with 

(25.2) li*||£p 
and any vector of the form b + db with 

(25.3) ||A||£r 

would be acceptable to the problem originator as substitutes for the specific 
data A and b. 

For comparison with these inherent error parameters p and p we may 
use Eq. (16.3) and (16.4) to define the following two functions of q: 

(25.4) p'(q) = (6m -3n + 41)n*'*\\A\\t} 

(25.5) p'(q) = (6m - 3n + 40>» || 6 H q 

Here we have replaced ||j 4||, of Eq. (16.3) by its upper bound n l/1 \\A\\. 

Ignoring the terms 0(q a ) in Eq. (16.3) and (16.4), we infer from Theorem 
(16.1) that if q is chosen small enough so that 

(25.6) p'(q) < p 

and 

(25.7) p'(q) < V 

then the solution computed using arithmetic of uniform precision q is the 
exact solution of some problem whose initial data differs from the given data 
by amounts satisfying Eq. (25.2) and (25.3). 

Similarly to select an q and at ^ q a for mixed precision computation one 
can use Eq. (17.13) and (17.14) to define 


( 25 . 8 ) 


p”(q) — 30w ,/a |M||q 
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and 

(25.9) — 29n || ft || 
and choose tj so that 

(25.10) *"(*) < 9 
and 

(25.11) v"(n)<v 

With this choice of q and o^ii’we infer from Theorem (17.11) that the 
computed solution is the exact solution of a perturbed problem with per* 
turbations smaller than the a priori uncertainty as described by Eq. (25.2) 
and (25.3). 

Recall that the multipliers of 11 A 11 if and 11 b 11 q in Eq. (25.4), (25.5), (25.8), 
and (25.9) were derived by considering worst-case propagation of computa¬ 
tional errors. In practice, replacing these multipliers by their square roots 
will generally give more realistic estimates of the norms of the virtual per¬ 
turbations due to the computational errors. 

We now turn to point 2. It will be convenient to formalize the notion of 
an "acceptably small” residual vector. Suppose we are willing to accept resid¬ 
ual vectors with norms as large as some number p. Then we may define the 
acceptable solution set as 

(25.12) *= {*:||(v! + E)x~(b + db) || ^ p, ||2?||£ p, \\db\\ £ ¥ ] 

It is important to note that the definition of the set X depends upon the three 
tolerance parameters, p, yr, and p, which should be chosen on the basis of 
actual knowledge about the problem and its data. 

Our purpose in writing Eq. (25.12) is not so much to establish this particu¬ 
lar definition of an acceptable solution set as it is to give some degree of 
concreteness to the general idea of an acceptable solution set and to identify 
some of the quantities that determine the "size” of this set. 

To assess the "size” of the set X we first observe that if Rank 04) < n, or 
if 9 and k (k denotes the condition number of A) are so large that xp/|| /||;> 
1, then A + £ will be singular for some ||£||<^p and the set X is unbounded. 

On the other hand, if Rank 04) = n and K9f\\A\\ < 1, then the per¬ 
turbation bound (9.13) or (9.14) may be used to obtain useful information 
regarding the diameter of the set X. The presence of the parameter p in the 
definition of the set X leads to some further possible increase in its size. 

If this set X is "large” in the sense that it contains vectors which are 
significantly different from each other, then some selection must be made of 
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a particular solution vector from X. This selection process can be viewed 
very broadly as including any steps that the problem originator might take 
to change the problem to one having a smaller solution set, generally con* 
tained in X. 

The criterion that one uses to reduce the size of the set X depends upon 
the application. A situation that occurs very commonly is one in which the 
problem Ax ^ b arises as a local linearization of a nonlinear least squares 
problem. This was mentioned in Chapter 1. In this case one is likely to prefer 
the use of the X e X having least norm in order to reduce the likelihood of 
leaving the region in which b — Ax is a good approximation to the nonlinear 
problem. 

Although there are many different motivations (statistical, mathematical, 
numerical, heuristic, etc.) that have been proposed for different specific pro¬ 
cedures for changing a given least squares problem, most of these procedures 
consist of performing one or more of the following four operations, not 
necessarily in the order listed. 

1. Left-multiply A and b by an m x m matrix G. 

2. Right-multiply A by an n x it matrix H with the corresponding change 
of variables x = Hx or x = Hx + 

3. Append additional rows to A and additional elements to b. 

4. Assign fixed values (often zero) to some components of the solution 
vector. This may be done either with respect to the original set of variables 
or a transformed set of variables. 

We shall expand on each of these four items in Sections 2,3,4, and 5, 
respectively. 

Finally in Section 6 we describe singular value analysis. By singular value 
analysis we mean the computation of a number of quantities derived from 
the singular value decomposition of the matrix A and the interpretation of 
these quantities as an aid in understanding the indeterminacies of the prob¬ 
lem Ax^b and in selecting a useful solution vector. The problem Ax — b to 
which singular value analysis is applied may of course be a problem resulting 
from preliminary application of various of the operations described in Sec¬ 
tions 2 to 5. 


Section 2. LEFT MULTIPLICATION OF A AND b 
BY A MATRIX O 

This operation changes the norm by which the size of the residual vector 
is assessed. Thus instead of seeking x to minimize (b — Ax) T {b — Ax), one 
changes the problem to that of minimizing (Gb — GAxf(Gb — GAx ), which 
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also can be written as (fc — Ax) r (G r GXb — Ax), or as (b — AxY W(b — Ax), 
where W — G r G. 

A commonly used special case is that in which the matrix G is diagonal. 
Then the matrix W is diagonal. In this case left-multiplication of the aug¬ 
mented matrix [A : i] by G can be interpreted as being a row-scaling opera¬ 
tion in which row i of [A: b) is multiplied by the number g„. 

Define 

(25.13) r = b — Ax 
and 

(25.14) f — Gr = Gb — GAx 
Then if G is diagonal the quantity to be minimized is 

(25.15) 11 HP 

Loosely speaking, assigning a relatively large weight, | g„ | (or equivalently 
w„) to the fth equation will tend to cause the resulting residual component 
| r, | to be smaller. Thus if some components of the data vector b, are known 
with more absolute accuracy than others, one may wish to introduce relatively 
larger weights, | g„ |, or w,„ in the >‘th row of [A : b). 

This procedure, in which G is diagonal, is generally referred to as weighted 
least squares. As a systematic scheme for assigning the weights suppose that 
one can associate with each component, b„ of the data vector b a positive 
number, a„ indicating the approximate size of the uncertainty in b,. If one 
has the appropriate statistical information about b it would be common to 
take a, to be the standard deviation of the uncertainty in b,. Then it would be 
usual to define weights by 



or equivalently 

Note that with this scaling all components of the modified vector S defined by 

= gttb< = ^f- 

have unit standard deviation. 

More generally, if one has sufficient statistical information about the 
uncertainty in b to place numerical values on the correlation between the 
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errors in different components of b, one may express this information as an 
m x m positive definite symmetric covariance matrix C. [See Plackett (1960) 
for a detailed discussion of these statistical concepts.] In this case one can 
compute the Cholesky factorization of C as described in Eq. (19.5) and (19.12) 
to (19.14) obtaining a lower triangular matrix Fthat satisfies 


C = FF T 

Then the weighting matrix G can be defined by 

G = F 

Operationally it is not necessary to compute the explicit inverse of F. 
Rather one would compute the weighted matrix [A : 6] by solving the equa¬ 
tions 

F[A:S] = [A:b] 

This process is straightforward because F is triangular. 

If G is derived from an a priori covariance matrix for the errors in b as 
described above, then the covariance matrix for the errors in the transformed 
vector 6 is the identity matrix. 

Whether on the basis of statistical arguments or otherwise, it is desirable 
to use a matrix G such that the uncertainty is of about the same size in all 
components of the transformed vector 6 ~ Gb. This has the effect of making 
the euclidean norm a reasonable measure of the “size” of the error vector db, 
as for instance in Eq. (25.3). 

Section 3. RIGHT MULTIPLICATION OF A BY A 
MATRIX H AND CHANGE OF 
VARIABLES x - Hi + % 

Hete one replaces the problem 


(25.16) 

AxSb 

by the problem 


(25.17) 

Ax~B 

where 


(25.18) 

A = AH 

(25.19) 

CM 

II 

1 

(25.20) 

x = Hx + { 
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The matrix H is n x l, with l<,n. The vector x is /-dimensional and the ma¬ 
trix X ism x /.If H is n x n diagonal, this transformation may be interpreted 
as a column scaling operation applied to A. 

If His it x n nonsingular, this transformation does not change the prob¬ 
lem mathematically. Thus the set of vectors satisfying x = Hx + £ where 
x minimizes \\6 — Xx\\ is the same as the set of vectors x minimizing 
\]b-Ax\\. 

However, unless H is orthogonal, the condition number of X will generally 
be different from that of A. Therefore, if one is using an algorithm, such as 
HFTI (14.9) or singular value analysis [Eq. (18.36) to (18.45)], that makes a 
determination of the pseudorank of A, the algorithm may make a different 
determination using X. 

Furthermore, if the pseudorank is determined to have a value k <n and 
one proceeds to compute the minimal length solution of the rank k problem, 
then the use of the transformation matrix H alters the norm by which the 
"size” of the solution vector is measured. This will, in general, lead to a 
different vector being selected as the "minimum length” solution. Thus in 
problem (25.16) the minimal length solution is a solution that minimizes ||x ||, 
whereas using the transformed problem in Eq. (25.17) the minimal length 
solution is a solution x of Eq. (25.17) that minimizes [|x||. This amounts to 
minimizing || H~ l {x — £)|| rather than minimizing ||x||. 

As to criteria for selecting H we first note that the use of the spectral norm 
of the perturbation matrix, E, in Eq. (25.2) is a realistic mathematical model 
only if the absolute uncertainties in different components of A are all of 
about the same magnitude. Thus, if one has estimates of the .uncertainty of 
individual elements of A, the matrix H can be selected as a column scaling 
matrix to balance the size of the uncertainties in the different columns of X. 

A somewhat similar idea can be based on a priori knowledge of the solu¬ 
tion vector x. Suppose it is known that the solution vector should be close to 
a known vector ( (the a priori expected value of x). Suppose further that one 
has an a priori estimate, e t , of the uncertainty of as an estimate of x,. One 
can take H to be the n x n diagonal matrix with diagonal components 

(25.21) h„ = a, 

Then the transformed variables 

(25.22) x, = Z-zJL 

07 

have unit a priori uncertainty and zero a priori expected values. 

More generally, if one has sufficient a priori statistical information to 
define an n x n positive-definite a priori covariance matrix K describing the 
uncertainty of £, then H can be computed as the upper triangular Cholesky 
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factor of C [see Eq. (19.16) to (19.18)): 
(25.23) C - HH T 


Then the a priori covariance of the transformed variable vector x of Eq. 
(25.20) is the n x n identity matrix, and the a priori expected value of x is the 
zero vector. 

The situation in which the separate components of x have approximately 
equal uncertainty is a desirable scaling of the variables if the problem is 
judged to have pseudorank k < n and a minimal length solution is to be 
computed. 

Since the condition number of X of Eq. (25.18) will, in general, be different 
from that of A, one may decide to choose H with the objective of making the 
condition number of X small. If A is nonsingular, there exists a matrix H 
such that Cond {AH) = 1. Such a matrix is given by H — R' 1 , where R is 
the triangular matrix resulting from Householder triangularization. One is 
not likely to have this matrix, R, available a priori. It is of interest to note, 
however, that if the data that lead to the determination of the a priori covar¬ 
iance matrix, C, of Eq. (25.23) were approximately the same as the present 
data [A : b ], then the matrix H of Eq. (25.23) will be approximately equal 
to A' 1 . Then Cond {AH) is likely to be fairly small. 

If no estimate of C is available, there remains the possibility of using a 
diagonal matrix H as a column scaling matrix to balance the euclidean 
norms (or any other selected norms) of the columns of X. This is accom¬ 
plished by setting 


(25.24) 


JM-* if|l«/ll*0 
1 I »IM-o 


where a, denotes the jth column vector of A. It has been proved by van der 
Sluis (1969) that with H defined by Eq. (25.24), using euclidean column 
norms, Cond {AH) does not exceed the minimal condition number obtainable 
by column scaling by more than a factor of n 112 . 

Improving the condition number has the advantage that perturbation 
bounds such as Eq. (9.10) will give less pessimistic results. It also may lead 
to a determination of pseudorank k = n, whereas without the improvement 
of the condition number by scaling a pseudorank of k < n might have been 
determined leading to unnecessary and inappropriate complications in ob¬ 
taining a solution. 

A transformation matrix H can be selected so that .Tor a submatrix of X 
has an especially convenient form, such as triangular or diagonal. Thus if 
one has computed the singular value decomposition, A = USV 1 ", then left- 
multiplying [A : b] by U T and right-multiplying A by H = V transforms A to 
the diagonal matrix S. Operationally one would not usually do this pre* and 
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postmultiplication explicitly as it would be done along with the computation 
of the singular value decomposition as described in Chapter 18. Systematic 
use of the singular value decomposition will be discussed in Section 6 of 
this chapter. 

Section 4. APPEND ADDITIONAL ROWS TO 

Here we discuss replacing the problem AxSbby the problem 

« km:] 

where F is an / x n matrix and d is an /-dimensional vector. 

Suppose one prefers that the solution vector x should be close to a known 
vector (. Setting F — I„ and d = { in (25.25) expresses this preference. In 
particular if d— { = 0 this expresses a preference for ||x|| to be small. 

The intensity of this preference can be indicated by a scaling parameter, 
say a, incorporated into the definition of F and d as 

F = <r‘/, and d=*o~'{ 

The number a can be regarded as an estimate of the size of the uncertainty 
in (. Thus setting a small causes the solution to be closer to 

A further refinement of this idea is to assign separate numbers, <r„ / =* 
1,..., n, where a, is an estimate of the size of the uncertainty of the ith com¬ 
ponent of Thus one would define d, — {Jo, for i — 1.» and let F be a 

diagonal matrix with diagonal elements f„ = <7f 1 • 

Finally, if one has sufficient a priori statistical information about the 
expected value, f, of the solution, x, to define a symmetric positive definite 
n x n covariance matrix, K, for (x — {)> then it is reasonable to set 

(25.26) F = L-* 

where L is the lower triangular Cholesky factor of K [see Eq. (19.12) to 


(19.14)] 


(25.27) 

K = LU 

and set 


(25.28) 

II 

*5 


Note that the case of d & 0 can be reduced to the case of d = 0 by a sim¬ 
ple translation if the system Fw = d is consistent. To see this let w be a solu- 
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tion of Fw — d and make the change of variables * ■= w + 5? in Eq. (25.25), 
which yields the transformed problem 

(25.29) 

It is important to make this change of variables if one intends to apply a 
minimal length solution method since the preference for ||x|| to be small is 
consistent with the conditions Fx s 0. 

We consider now the question of the relative weighting of the two sets of 
conditions Axsb and Fxszd in Eq. (25.25). From a formal statistical point 
of view one might say that the appropriate relative weighting is established 
by setting up the problem in the form 

« to a-cw: a-K] 

where {G r G)~ l is the a priori covariance matrix of the uncertainty in the data 
vector b and (F r F)~ l is the a priori covariance matrix of the uncertainty in 
the a priori expected value { of the solution vector x. In practice, however, 
these a priori covariance matrices, particularly (F T F)~ 1 , may not be known 
with much certainty and one may wish to explore the changes that different 
relative weighting produces in the solution vector and in the residual vector. 
For this purpose introduce a nonnegative scalar weighting parameter X 


into problem (25.30) and consider the new problem 

(25.31) 

DM 

where 


(25.32) 

I=GA 

(25.33) 

6— Gb 

and 


(25.34) 

ft* 

II 

*5 



For the convenience of readers who may be more accustomed to other 
ways of motivating and stating the least squares problem involving a priori 
covariance matrices we note that problem (25.51) is the problem of finding 
a vector x to minimize the quadratic form 
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or equivalently 

(Ax - b) T (G r G)(Ax ~b) + X*(x - ® 

The idea of using a relative weighting parameter X in this context was 
discussed by Levenberg (1944). Further elaboration and applications of the 
technique are given by Hoerl (1959,1962, and 1964), Morrison (1960), 
Marquardt (1963 and 1970), and Hoerl and Kennard (1970a and 1970b). 
As a result of the 1963 paper and a computer program contributed to the 
computer information exchange organization SHARE by Marquardt, the 
use of this idea in problems arising from the linearization of nonlinear least 
squares is often called Marquardt's method. The study of problem (25.31) as a 
function of X has also been called ridge regression or damped least squares. 

To analyze the dependence of the solution vector and the residual vector 
in problem (25.31) upon the parameter X we first introduce the change of 
variables 


(25.35) 


x = ( + F~ l y 


and obtain the transformed problem 


(25.36) 

where 

and 


GM5] 

A - lF-« 
t~S-M 


The translation by { and scaling by F~ l used in Eq. (25.35) have been 
discussed in Section 3 of this chapter. Their purpose is to produce the new 
variable y that is better scaled for meaningful assessment of the “size” of the 
solution vector. 

Write a singular value decomposition of A: 


(25.37) 



Recall that S = Diag fo,..., $„}. If Rank (A) = k< n, then s, = 0 for 
i > k. Introduce the orthogonal change of variables 


(25.38) y = Vp 

and left-multiply Eq. (25.36) by the orthogonal matrix 
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obtaining the new least squares problem 


(25.39) 

where 

(25.40) 




Finally, for X > 0, the submatrix, XI„ in problem (25.39) can be elimi¬ 
nated by means of left-multiplication by appropriate Givens rotation ma¬ 
trices. Thus to eliminate the ith diagonal element of XI„ left-multiply Eq. 
(25.39) by a matrix that differs from an (m + n)th-order identity matrix only 
in the following four positions: 

Col. / Col. m + i 


Row / ... w ••• 75 


Row #»! + /••• — 


(si Tl PF 5 


S, 

TsFTW 7 * 


• * • 


After this elimination operation for i = 1,..., n the resulting equivalent 
problem is 

SB. 1 _ um 

s ,r* u HJ w 


(25.41) 


where 

(25.42) 

(25.43) 

and 


*»*« 

L J 


h«'\)n 


= -f A 1 ) 1 ' 2 

g, l = n + 1,..., m 


A,u> - (sf + 




S<» = Diag {^>.*«>} 
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with 


(25.44) *,«> = (sf + X 2 ) 1 ' 2 I - 1,.... h 


For 2 = 0 problem (25.39) has a diagonal matrix of rank k and the solu¬ 
tion vector p w is given by 


(25.45) 



i = l.i: 


n 


For X > 0 we use Eq. (25.41) to (25.44) and obtain the solution compo¬ 
nents 


(25.46) = 




t — 

i = k + 


Furthermore we have 


(25.47) 



Note that Eq. (25.46) and (25.47) remain valid for 2 = 0. 

The vector p M , which is a solution for problem (25.39), can be transformed 
by the linear equations (25.38) and (25.35) to obtain vectors y w and x M , 
which are solutions for problems (25.36) and (25.31), respectively. 

We are primarily interested in the problem Ax~B and have introduced 
problem (25.31) only as a technical device for generating solutions to a par¬ 
ticular family of related problems. Thus formulas will also be derived for the 
quantity to* — ||5 — &x w ||. With 2 ;> 0 and p w defined by Eq. (25.46) we 
obtain 


(25.48) 




\\6-Xx^\\ 2 = \\i-Ay^\\ 2 

-cw 

=It + JE . e 


Note that as 2 increases, 11 p w 11 decreases and (o x increases. Thus the prob¬ 
lem originator has the opportunity of selecting 2 to obtain some acceptable 
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compromise between the size of the solution vector p a> and the size of the 
residua] norm a x . 

The set of solutions of problem (25.31) given by this technique has an 
important optimality property expressed by the following theorem. 

(25.49) Theorem [Morrison {I960); Marqueurdt {1963)) 

For a fixed nonnegative value of 2, say, X, let y be the solution 
vector for problem (25.36), and let d> = ||6 — Ay||. Then <5 is the 
minimum value qf||6 — Ay || for all vectors y satisfy «g||y||<£||y||. 

Proof: Assume the contrary. Then there exists a vector y with 
IlyllrSIlyll satisfying \\i - Ay\\ <\\h - Ay\\. Therefore, \\t> — Ay\\* + 
JT*||y || 2 < j|£ — Ay |P + J[*||y||, which contradicts the assumption that y is 
the least squares solution of problem (25.36) and thus minimizes || t — Ay |p 

+ X 1 II ^ IP* 

A simple tabular or graphic presentation of the quantities || p w || and o> A 
given by Eq. (25.47) and (25.48) can be very useful in studying a particular 
least squares problem. Further detail can be obtained by tabular or graphic 
presentation of the individual solution components [Eq. (25.46), (25.38), and 
(25.35)] as functions of 2. An example of this type of analysis is given in 
Chapter 26. 

One may wish to solve Eq. (25.47) or (25.48) to find a value of 2 which 
produces a prespecified solution norm or residual norm. If Newton’s method 
is used for this purpose the following expressions will be useful: 

t = X 2 

- fh 

ii ip=£ m 

v - -k m 


and 



Viu) = 



•*=£*««>+. .£;,«? 



194 PRACTICAL ANALYSIS OF LEAST SQUARES PROBLEMS 


CHAP. 25 


If one solves problem (25.36) directly, using any algorithm that determines 
the norm, say p x , of the residual vector, then it should be noted that p x sat* 
isfies 

(25.50) + IP 

Thus if one wishes to obtain the quantity a> x , this can be done by computing 
||yU>||3 aa d then solving Eq. (25.50) for a x . 


Section 6. DELETING VARIABLES 

Deleting a variable from a problem is equivalent to fixing the value of 
that variable at zero. If one variable, say x„ is deleted, then the problem 

(25.51) Axsb 
is changed to 

(25.52) Axgzb 

where A is the m x (n — 1) matrix that consists of the first n — 1 column 
vectors of A and x is an (n — l)*dimensional vector. 

Assuming m ;> «, the separation theorem (5.12) implies that the singular 
values of A interlace with those of A, from which it follows that Cond (A) <, 
Cond (A). By repeated application of this process it follows that removing 
any proper subset of the variables leads to a matrix whose condition number 
is no larger than that of the original matrix. 

Clearly the minimum residual obtainable in problem (25.52) is not smaller 
than the minimum residual attainable in problem (25.51). 

Thus like some of the techniques discussed previously, removing variables 
is a device that reduces, or at least does not increase, the condition number 
of a matrix at the expense of increasing, or at least not decreasing, the norm 
of the residual vector. 

Although we feel it is useful for the purpose of comparing variable 
deletion or variable augmentation with other stabilization methods to note 
the properties stated above, it is nevertheless true that these properties are 
not usually taken as the motivating concepts for this type of procedure. More 
commonly one uses variable deletion or augmentation because one wishes 
to find the smallest number of parameters that can be used in the solution 
and still give an acceptably small residual norm. 

In some cases there is a natural ordering of the variables, for instance, 
when the variables are coefficients of a polynomial in one variable. Then it is 
a straightforward matter to obtain a sequence of solutions, first using only 
the first variable, next the first and second variable, and so forth. In fact, in 
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the special case of fitting data by a polynomial or by truncated Fourier series 
there are quite specialized algorithms available [e.g., Forsythe (1937)]. 

If there is not a natural ordering of the variables, then the following prob¬ 
lem of subset selection is sometimes considered. For each value of k = 1,2, 
..., n, find the set /* consisting of k indices such that the residual norm p k 
obtained by solving for only the k variables x„ i e J k , is as small as can be 
attained by any set of k variables. Usually one would also introduce a linear 
independence tolerance x and limit consideration to sets of k variables such 
that the associated matrix satisfies some conditioning test (such as size of 
pivot element) involving r. 

The sequence [p k } is obviously nonincreasing with increasing k and one 
might introduce a number p with the meaning that the computational pro¬ 
cedure is to stop if a value of k is reached such that p k < fi. Alternatively, 
one might stop when p k — p k+l is smaller than some tolerance, possibly de¬ 
pending upon k. This latter type of termination test arises from statistical 
considerations involving the “F" distribution [e.g., see Plackett (I960]. 

This problem, as stated, appears to require for each value of k an ex¬ 


haustive procedure for solving for all M combinations of k variables. This 
exhaustive approach becomes prohibitively expensive if ^ is large. 


Methods have been devised that exploit the partial ordering (of set inclu¬ 
sion) among the subsets to determine the optimal subsets J k without explic¬ 
itly testing every subset. See LaMotte and Hocking (1970) and Garside 
(1971) and references cited there for an exposition of these ideas. Sugges¬ 
tions for basing subset selection on ridge regression (see Section 4, this 
chapter) are given in Hoerl and Kennard (1970b). 

An alternative course of action is to compromise by solving the following 
more restricted problem instead. Solve the problem as stated above fork = 1. 
Let /, be the solution set After the set J k has been determined, consider 
only candidate sets of the form J k U [j] for j $ J k in searching for a pre¬ 
ferred set of k -{- 1 variables that will constitute the set /*+,. Let us denote 
the sequence of sets formed in this way by /*, k = 1 ,...,«, and the associ¬ 
ated residual norms by p k . Note that J k = J k and p k = p k for k = 1 and k — 
«, but for 1 < k < n, J k is generally not the same as J k , and p k ^ p k . 

This type of algorithm is referred to as stepwise regression [Efroymson, 
pp. 191-203 in Ralston and Wilf (I960)}. The computation can be organized 
so that the selection of the new variable to be introduced at stage k is only 
slightly more involved than the pivot selection algorithm normally used in 
solving a system of linear equations. 

This idea may be elaborated further to consider at each stage the possibil¬ 
ity of removing variables whose contribution to making the residual small 
has become insignificant due to the effect of variables subsequently introduced 
into the solution (Efroymson, be. cit.). 
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The primary mathematical complication of stepwise regression is the fact 
that the amount that one coefficient, say x } , contributes to reducing the re¬ 
sidual norm depends in general upon what other set of coefficients are being 
included in the solution at the same time. This drawback can be circumvented 
by performing a linear change of variables to obtain a new set of variables 
whose individual effects upon the residual vector are mutually independent. 
In statistical terminology the new set of variables is uncorrelated. In algebraic 
terms this amounts to replacing the A matrix by a new matrix X = AC such 
that the columns of ^ are orthogonal. There are a variety of distinct matrices, 
C, that would accomplish such a transformation and in general different sets 
of uncorrelated variables would be produced by different such matrices. One 
such transformation matrix that has some additional desirable properties is 
C—V where V is the matrix arising in a singular value decomposition, A — 
VSV T . 

In particular, if the transformation matrix C is required to be orthogonal, 
then for AC to have orthogonal columns it follows that Cmust be the matrix 
V of some singular value decomposition of A. 

Furthermore the numbers t, that occur as the square roots of the diagonal 
elements of the diagonal matrix [(AC) T (AQ]~ l have a statistical interpretation 
as being proportional to the standard deviations of the new coefficients. The 
choice of C—V causes these numbers t, to be reciprocals of the singular 
values of A. This choice minimizes the smallest of the standard deviations, 
and also minimizes each of the other standard deviations subject to having 
selected the proceeding variables. 

The use of the singular value decomposition of A will be discussed further 
in the next section. 


Section 6. SINGULAR VALUE ANALYSIS 


Suppose a singular value decomposition (see Chapters 4 and 18) is com' 
puted for the matrix A. 

(25.53) A = 

One can then compute 

(25.54) g = U T b 
and consider the least squares problem 



(25.55) 
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where p is related to x by the orthogonal linear transformation 


(25.56) 


x^Vp 


Problem (25.55) is equivalent to the problem Ax S b in the sense discussed 
in Chapter 2 for general orthogonal transformations of least squares prob¬ 
lems. 

Since 5 is diagonal (5 = Diag {s u ..., s„}) the effect of each component 
of p upon the residual norm is immediately obvious. Introducing a compo¬ 
nent Pj with the value 

(25.57) A = f 

s j 


reduces the sum of squares of residuals by the amount gj. 

Assume the singular values are ordered so that s k i j* + i , k — 1. 

n — 1. It is then natural to consider “candidate” solutions for problem (25.55) 
of the form 



(25.58) 



k * 0,1,..., n 



where p, is given by Eq. (25.57). The candidate solution vector p tk> is the 
pseudoinverse solution (i.e., the minimal length solution) of problem (25.S5) 
under the assumption that the singular values s, for j > k are regarded as 
being zero. 

From the candidate solution vectors p {k) one obtains candidate solution 
vectors x tk) for the problem Ax S b as 

(25.59) x‘» = Vp M = pjv'» k = 0 . n 

where denotes the /th column vector of V. Note that 

(25.60) ii *<*> ||* - \\ P *> ii*=I; pj - g (&)’ 

hence \\x ik) \\ is a nondecreasing function of k. The squared residual norm 
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associated with x M is given by 

(25.61) pi = ||6 — Ax™ |P - £ t gj 


Inspection of the columns of the matrix V associated with small singular 
values is a very effective technique for identifying the sets of columns of A 
that are nearly linearly dependent [see Eq. (12.23) and (12.24)]. 

It has been our experience that computing and displaying the matrix V, 
the quantities s k , si 1 , p k , x™, and || x™ || for k = and g k , gl, pi, and 

p k for k — 0,1. n is extremely helpful in analyzing difficult practical 

least squares problems. For certain statistical interpretations [see Eq. (12.2)] 
the quantities 


(25.62) 



Pi 


1/2 


k — 0,1,.... n 


are also of interest. 

Suppose the matrix A is ill-conditioned; then some of the later singular 
values are significantly smaller than the earlier ones. In such a case some of 
the later p, values may be undesirably large. Typically one hopes to locate 
an index k such that all coefficients p, for j <, k are acceptably small, all 
singular values s, for j <.k are acceptably large, and the residual norm p k is 
acceptably small. If such an index k exists, then one can take the candidate 
vector x™ as an acceptable solution vector. 

This technique has been used successfully in a variety of applications. For 
example, see Hanson (1971) for applications to the numerical solution of 
Fredholm integral equations of the first kind. 

Once the singular values and the vector g have been computed it is also 
a simple matter to compute the numbers ||p w) || and co k of the Levenberg- 
Marquardt stabilization method [see Eq. (25.47) and (25.48)] for a range of 
values of A. These quantities are of interest due to the optimality property 
stated in Theorem (25.49). 

An example in which ail the quantities mentioned here are computed and 
interpreted for a particular set of data is given in the following chapter. 



EXAMPLES OF SOME METHODS OF 
A Jr ANALYZING A LEAST SQUARES 

jCO problem 


Consider the least squares problem Ax ^ b where the data matrix 
[A tSxt : b lSxi ] is given by Table 26.1. (These data are given also as the data 
element, DATA4, in Appendix C.) We shall assume there is uncertainty of 
the order of 0.5 x 10'* in the elements of A and 0.S x 10' 4 in the compo¬ 
nents of b. 

First consider a singular value analysis of this problem as described in 
Section 6 of Chapter 25. The computation and printing of the quantities used 
in singular value analysis is accomplished by the Fortran subroutine SVA 
(Appendix C). The Fortran main program PROG4 (Appendix C) applies the 
subroutine SVA to the particular data of the present example. Executing 
PROG4 on a UNIVAC 1108 computer resulted in the output reproduced in 
Fig. 26.1. 

The subroutine SVA in general performs a change of variables x— Dy 
and treats the problem (AD)y s b. In this example we have set D — /so the 
label y in Fig. 26.1 is synonymous with x for this example. 

Recall that the vector g (output column headed H G COEF”) is computed 
as g = U T b, where U is orthogonal. Since the fourth and fifth components of 
g are smaller than the assumed uncertainty in b, we are willing to treat these 
two components of g as zero. This leads us to regard the third candidate 
solution, jc < 3) (output column headed u SOLN 3”) as the most satisfactory 
solution. 

Alternatively one could compare the numbers a k [see Eq. (25.62)] in the 
column headed “N.S.R.C.S.S” (meaning formalized Square Root of Cumu¬ 
lative Sum of Squares) with the assumed uncertainty (0.5 X 10~ 4 ) in b. Note 
thatff 2 (= 1.1107 x 10~ a ) is significantly larger than this assumed uncertainty 
while o, (= 4.0548 x 10 s ) is slightly smaller than this uncertainty. This 
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Table 26.1 the data matrix [A : 6] 


—.13405547 

—.20162827 

-.16930778 

-.18971990 

-.17387234 

-.4361 

—.10379475 

-.15766336 

-.13346256 

-.14848550 

-.13597690 

-.3437 

-.08779597 

-.12883867 

-.10683007 

-.12011796 

-.10932972 

-.2657 

.02058554 

.00335331 

-.01641270 

.00078606 

.00271659 

-.0392 

-.03248093 

-.01876799 

.00410639 

-.01405894 

-.01384391 

.0193 

.05967562 

.06667714 

.04352153 

.05740438 

.05024962 

.0747 

.06712457 

.07352437 

.04489770 

.06471862 

.05876455 

.0935 

.08687186 

.09368296 

.05672327 

.08141043 

.07302320 

.1079 

.02149662 

.06222662 

.07213486 

.06200069 

.05570931 

.1930 

.06687407 

.10344506 

.09153849 

.09508223 

.08393667 

.2058 

.15879069 

.18088339 

.11540692 

.16160727 

.14796479 

.2606 

.17642887 

.20361830 

.13057860 

.18385729 

.17005549 

.3142 

.11414080 

.17259611 

.14816471 

.16007466 

.14374096 

.3529 

.07846038 

.14669563 

.14365800 

.14003842 

.12571177 

.3615 

.10803175 

.16994623 

.14971519 

.15885312 

.14301547 

.3647 


could be taken as a reason to select x**’ as the preferred candidate 
solution. 

If the assumed uncertainty in b is given the statistical interpretation of 
being the standard deviation of the errors in b, then multiplication of this 
quantity (0.S x 10~ 4 ) times each reciprocal singular value (column headed 
“RECIP. S.V.”) gives the standard deviation of the corresponding component 
of the vector p (column headed “P COEF”). Thus the first three components 
of p exceed their respective standard deviations in magnitude while the last 
two components are smaller in magnitude than their standard deviations. 
This could be taken as a reason to prefer the candidate solution x (,> . 

As still another method for choice of an x‘*\ one would probably reject 
the candidate solutions x <u and x (I> because their associated residual norms 
are too large (column headed “RNORM”). One would probably reject x ( ” 
because ||x <5, ||is too large (column headed “YNORM”). The choice between 
x <s » and x <4) is less decisive. The vector x <3) would probably be preferred to 
x <4) since || x <J) || < ||x <4) || and the residual norm associated with x ,4) is only 
slightly smaller than that associated with x tv . 

While these four methods of choosing a preferred x (k) have led us to the 
same selection, x (J) , this would not always be the case for other sets of data. 
The user must decide which of these criteria (possibly others included) are 
most meaningful for his application. 

It is of interest to compare some other methods of obtaining stabilized 
solutions to this same problem. The Levenberg-Marquardt analysis (Chap¬ 
ter 25, Section 4) provides a continuum of candidate solutions. Using the 
Levenberg-Marquardt data shown in Fig. 26.1 one can produce the graph, 
Fig. 26.2, of RNORM versus YNORM. Following Theorem (25.49) this 
curve constitutes a boundary line in the (YNORM-RNORM)-plane such 
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Fig. 26.1 Output from the subroutine SVA for the sample problem 
of Chapter 26. 


that for any vector y the coordinate pair Q|y||, ||A — Ay ||) lies on or above 
the curve. 

For more detailed information Eq. (25.46), (25.38), and (25.35) can be 
used to compute and plot values of the individual solution components as 
functions of A. Figures 26.3 and 26.4 show this information for the present 
example. Graphs of this type are extensively discussed in Hoerl and Kennard 
(1970b). 

Figure 26.5 provides a comparison of solution norms and residual norms 
of the five candidate solutions obtained by singular value analysis with the 
corresponding data for the continuum of Levenberg-Marquardt solutions. 

We have also applied the subroutine, HFTI (see Appendix C) to this 
example. The magnitudes of the diagonal elements of the triangular matrix 
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Fig. 26.2 Residual norm versus solution norm for a range of values 
of the Levenberg-Marquardt stabilization parameter A. 
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Fig. 26.3 Sol ution confidents and residual norm as a function of A. 
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X 

Fig. 26A Same data as Fig. 26.3 with expanded vertical scale. 

R to which the matrix A is transformed are 0.52,0.71 x 10~*, 0.91 x 10“ 2 , 
0.14 X 10“ 4 , and 0.20 x 10~*. It is interesting to note that these numbers 
agree with the respective singular values of A to within a factor of two. From 
Theorem (6.31) it is known that the singular values, s h and the diagonal 
elements r,i must satisfy 

1.00 ii. ^ 2.24 

r it 

0.58 ^ i*- ^ 2.00 
'n 

0.33 ^ ^ ^ 1.73 

0.18 ^ ii- ^ 1.41 

0.09 1.00 

By setting the pseudorank tolerance parameter successively to the values 
t = 0.29,0.040,0.0046,0.0000073, and 0.0, the subroutine HFTi was used 
to produce five different solutions, say z (t> , associated with pseudoranks of 
k = 1,2,3,4, and 5. The solution norms and residual norms for these five 
vectors are shown in Table 26.2. 
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Fig. 2 63 Candidate solutions x<‘> from singular value analysis 
compared with the continuum of Leveeberg-Marquardt solutions. 


Table 26.2 solution and residual norms usino HFTI on thb sample 

PROBLEM 


k 



1 

0.99719 

0.216865 

2 

2.24495 

0.039281 

3 

4.58680 

0.000139 

4 

4.92951 

0.000139 

5 

220.89008 

0.000138 


Note that the data of Table 26.2 are quite similar to the corresponding 
data (columns headed “YNORM” and “RNORM") given in Fig. 26.1 for 
the candidate solutions obtained by singular value analysis. 

As still another way of analyzing this problem, solutions were computed 
using each of the 31 possible nonnull subsets of the five columns of the 
matrix A. Solution norms and residual norms for each of these 31 solutions 
are listed in Table 26.3 and are plotted in Fig. 26.6. 



NORM OF SOLUTION VECTOR 


Fig. 26.6 Solutions using subsets of the columns of A. 


We shall use the notation to denote the solution obtained using 
columns i,j ,.... 

From Fig. 26.6 we see that the simplest form of stepwise regression would 
select solution w 0 ’ as the preferred solution among those involving only one 
column. 

It would next consider solutions w <, ' 3, > w a>,> , H' (3 • 4, , and w (3 - 5) . It would 
select w a ’ 3> as the one of these four solutions providing the smallest residual 
norm. Note that this solution has a residual norm larger by a factor of 85 
than that corresponding to w (l * 3> . This latter solution clearly gives the smallest 
norm obtainable using any set of two columns. 

At the next stage simple stepwise regression would select tv"' 3 ' 3 ’, which 
is quite similar in solution norm and residual norm to the previously men¬ 
tioned x (v and z (i> . 
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The progress of a stepwise regression algorithm beyond this stage would 
depend critically on details of the particular algorithm due to the very ill- 
conditioned problems that would be encountered in considering sets of four 
or five columns. 

Table 26.3 solution and residual norms using subsets of the columns 


Columns Used 

11 Nil 

1(6 — ilw|| 

i 

2.46 

0.40 

2 

1.92 

0.22 

3 

2.42 

0.07 

4 

2.09 

0.19 

5 

2.30 

0.19 

1.2 

5.09 

0.039 

1,3 

2.72 

0.052 

1,4 

4.53 

0.023 

1.5 

5.07 

0.001 

2.3 

3.03 

0.053 

2.4 

20.27 

0.030 

2,5 

17.06 

0.128 

3,4 

3.07 

0.056 

3,5 

2.97 

0.058 

4,5 

17.05 

0.175 

1,2,3 

22.1 

0.00018 

1.2.4 

10.8 

0.00018 

1,2,5 

5.0 

0.00014 

1,3,4 

8.1 

0.00015 

1,3,5 

5.0 

0.00014 

1,4,5 

4.9 

0.00014 

2,3,4 

13.5 

0.00020 

2,3,5 

7.6 

0.00014 

2,4,5 

24.0 

0.00028 

3,4,5 

17.3 

0.00017 

1,2,3,4 

10.3 

0.00014 

1,2,3,5 

5.0 

0.00014 

1,2,4,5 

5.0 

0.00014 

1,3,4,5 

5.0 

0.00014 

2,3,4,5 

9.0 

0.00014 

1.2,3,4,5 

220.9 

0.00014 




MODIFYING A QR DECOMPOSITION 
TO ADD OR REMOVE ROW VECTORS 
WITH APPLICATION TO SEQUENTIAL 
PROCESSING OF PROBLEMS 
HAVING A LARGE OR BANDED 
A# COEFFICIENT MATRIX 


Section 1. Sequential Accumulation 

Section 2. Sequential Accumulation off Banded Matrices 

Section 3 . An Example: Line Splines 

Section 4. Data Fitting Using Cubic Spline Functions 

Sections. Removing Rowe off Data 


In this chapter we present adaptations of orthogonal transformations to 
the sequential processing of data for Problem LS. Such methods provide a 
means of conserving computer storage for certain types of problems having 
voluminous data. 

These methods also have important application to problems in which one 
wishes to obtain a sequence of solutions for a data set to which data are 
being sequentially-added or deleted. A requirement for this type of computa¬ 
tion arises in the class of problems called sequential estimation, filtering, or 
process identification. The augmentation methods presented in this chapter 
have excellent properties of numerical stability. This is in contrast to many 
published approaches to this problem that are based on the notion of updat¬ 
ing the inverse of a matrix. 

The problem of deleting data can be an inherently unstable operation 
depending upon the relationship of the rows to be deleted to the whole data 
set. Some methods will be presented that attempt to avoid introducing any 
additional unnecessary numerical instability. 

Write the least squares problem in the usual way: 

(27.1) Ax = b, A mxm , b m xi 

In Section 1 we treat the problem in which the number of rows of A is 
large compared with the number of columns. In Section 2 attention is devoted 
to the case in which A has a “banded” structure. The methods presented in 
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both Sections 1 and 2 are applicable to the sequential estimation problem since 
the solution vector or its associated covariance matrix can readily be com¬ 
puted at each step of the sequential matrix accumulation if required. In 
Section 3 an application of these ideas is presented using the example of curve 
fitting using line splines. In Section 4 a procedure is described for least squares 
data fitting by cubic splines having equally spaced breakpoints. This provides 
a further example of the applicability of the banded sequential accumulation 
method of Section 2. 

Methods of deleting data will be described in Section S. 


Section 1. SEQUENTIAL ACCUMULATION 


In this section we shall describe an algorithm for transforming the matrix 
[A : b ] to upper triangular form without requiring that the entire matrix 
[A: b] be in computer storage at one time. A formal description of this pro¬ 
cedure will be given as Algorithm SEQHT (27.10). 

We first establish the notation to be used and give an informal description 
of the process. Write the matrix A and the vector b in partitioned form: 


(27.2) 


(27.3) 



where each A, is m, x it and each b, is a vector of length m,. We have m — 
m, -f • • ■ + m„ of course. The integers m, can be as small as 1, which per¬ 
mits the greatest economy of storage. Remarks on the dependence of the total 
number of arithmetic operations on the size of m, are given on page 211. 

The algorithm will construct a sequence of triangular matrices [A,: d t ], 
i = 1,..., q, with the property that the least squares problem 


Rpc~d, 


has the same solution set and the same residual norm as the problem 
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The significant point permitting a saving of storage is the fact that for each i, 
the matrix [it,: dj can be constructed and stored in the storage space previ¬ 
ously occupied by the augmented matrix 

(27.4) 


Thus the maximum number of rows of storage required is maxj {m, + min 

[» + 1, E/-i 

For notational convenience let [/?„: rf 0 ] denote a (null) matrix having no 
rows. At the beginning of the /th stage of the algorithm one has the /ft,., x 
(n + 1) matrix [/l,-,: d t .,] from the (/ — l)st stage and the new m, x 
(n + 1) data matrix [A,: fcj. Let m, ~ + m,. Form the m, x (n + 1) 

augmented matrix of Eq. (27.4) and reduce this matrix to triangular form 
by Householder triangularization as follows: 



(27.5) 



Ri'df }ifi| 

. 0 : 0 . }m, — ft, 


where th, — min {« + 1, m,}. 

This completes the fth stage of the algorithm. For future reference, let 
the result at the fth stage of the algorithm be denoted by 


(27.6) 



It is easily verified that there exists an orthogonal matrix Q such that 


QlA:b] 


R d' 
0 e 
0 0 


Therefore, the least squares problem 
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is equivalent to the original problem in Eq. (27.1) in that it has the same set 
of solution vectors and the same minimal residual norm. Furthermore the 
matrix R has the same set of singular values as the matrix A of Eq. (27.1). 

We shall now describe a computing algorithm that formalizes this pro* 
cedure. To this end set 

(0 if j = 0 

(27.8) = \j ... 

[2 m, if j > 0 

and 

(27.9) ft = max {m, + min [n + 1, v,.,]} 

ISIS* 

The processing will take place in a computer storage array W having at 
least ft rows and n + 1 columns. We shall use the notation W(/,: i 1 ,j i :j t ) 
to denote the subarray of W consisting of rows i t through i 2 of columns j t 
through j 2 . We use the notation W(i,j) to denote the (ij) element ofthearray 
IV. The symbol p identifies a single storage location. 

(27.10) Algorithm SEQHT (Sequential Householder Triangularization ) 

Step Description 

1 Set /0. 

2 For / := 1,..., q, do Steps 3-6. 

3 Set p := / 4- m,. 

4 Set W(l + 1 :p, 1: it + 1) := \A, : *,] [see Eq. (27.2) and 
(27.3)]. 

5 For i := 1.min (n + 1 ,p — 1), execute Algorithm 

HI (i, max (/ + 1, / + 1), p, W( 1, /), p, W( 1, i + 1), 
n — i 4-1). 

6 Set /min (« + 1 ,p). 

7 Remark: The matrix [A : b] has been reduced to upper 
triangular form as given in Eq. (27.6). 

A sequence of Fortran statements implementing Step S of this algorithm 
is given as an example in the user’s guide for the subroutine HI 2 in Appen¬ 
dix C. 

Notice, in Step 4, that only the submatrix [A,: b t ] needs to be intro¬ 
duced into computer stores at one time. All these data can be processed in 
the p x (« + 1) working array W. By more complicated programming one 
could further reduce the storage required by exploiting the fact that each ma¬ 
trix [R } : dj] is upper triangular. 
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For the purpose of discussing operation counts let a denote an addition 
or subtraction and ft denote a multiplication or division. Define 

iwn* — n*/3 
- - T^ 

It was noted in Table 19.1 that if quadratic and lower-order terms in m and 
n are neglected, the number of operations required to triangularize anmxn 
matrix (m > n) using Householder transformations is approximately v(2a + 
2 ju). If the Householder processing is done sequentially involving q stages as 
in Algorithm SEQHT (27.10), then the operation count is increased to ap¬ 
proximately v(2« + 2 n)(m + q)fm. If the entering blocks of data each con¬ 
tain k rows ( kq — m), then the operation count can be written as v(2a + 2 ft) 
(k + l)/k. 

Thus sequential Householder accumulation increases in cost as the block 
size is decreased. In the extreme case of a block size of k — 1, the operation 
count is approximately doubled relative to the count for nonsequential 
Householder processing. 

For small block sizes it may be more economical to replace the House¬ 
holder method used at Step S of Algorithm SEQHT (27.10) with one of the 
methods based on 2 x 2 rotations or reflections. The operation counts for 
these methods are independent of the block size. The operation count for 
triangularization using the Givens method [Algorithms G1 (10.25) and G2 
(10.26)] is v(2« + 4/t). The 2x2 Householder transformation expressed 
as in Eq. (10.27) has an operation count of v(3a + 3 ft). 

Gentleman’s modification (see Chapter 10) of the Givens method reduces 
the count to v(2* + 2 ft). Thus this method is competitive with the standard 
Householder method for nonsequential processing and requires fewer arith¬ 
metic operations than the Householder method for sequential processing. 

Actual comparative performance of computer programs based on any of 
the methods will depend! strongly on coding details. 

Following an application of Algorithm SEQHT (27.10), if the upper trian¬ 
gular matrix R of Eq. (27.6) is nonsingular, we may compute the solution, X, 
by solving 

(27.11) Rx = d 
The number e in Eq. (27.6) will satisfy 

(27.12) 1 e| = \\AX — 6|| 

In many applications one needs the unsealed covariance matrix [see Eq. 

( 12 . 1 )] 


(27.13) 


C = {jPA)-' = r-'(r-'Y 
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Computation of R ~ 1 can replace R in storage and then computation of 
(the upper triangular part of) can replace R~ l in storage. Thus the 

matrix C of Eq. (27.13) can be computed with essentially no additional 
storage. See Chapter 12 for a more detailed discussion of this computation. 
Furthermore, the quantity a 1 of Eq. (12.2) can be computed as 


m — n 

where e is defined by Eq. (27.6). 

For definiteness. Algorithm SEQHT was presented aB a loop for t — 1, 
..., q. In an actual application using this sequential processing it is more 
likely that Steps 3-6 of the algorithm would be implemented as a subroutine. 
The number q and the total data set represented by the matrix [i4:6] need 
not be known initially. The calling program can use the current triangular 
matrix at any stage at which R is nonsingular to solve for a solution based 
on the data that have been accumulated up to that point. By using an 
additional set of n(n + 1 )/2 storage locations, the calling program can also 
compute the upper triangular part of the unsealed covariance matrix C, — 
R? , (R? i ) T at any stage at which R, is nonsingular. Thus Steps 3-6 of Algo¬ 
rithm SEQHT provide the algorithmic core around which programs for 
sequential estimation can be constructed. 

Section 2. SEQUENTIAL ACCUMULATION 
OF BANDED MATRICES 

In some problems, possibly after preliminary interchange of rows or 
columns, the data matrix [A: 6] of Eq. (27.2) and (27.3) has a banded struc¬ 
ture in the following sense. There exists an integer n t <,n and a nondecreas¬ 
ing set of integers j t . j t such that all nonzero elements in the submatrix 

A, occur in columns j, through j, -f n b — 1. 

Thus A, is of the form 


j, — 1 n b n - — j, + 1 

(27.14) A, = [ 0 0 \\m, 

We shall refer to n h as the bandwidth of A. 

It can be easily verified that all nonzero elements in the ith row of the up¬ 
per triangular matrix R of Eq. (27.6) will occur in column i through / + n* — 
1. Furthermore, rows 1 through j, — 1 of R will not be affected by the pro¬ 
cessing of the submatrices A„... ,A f in Algorithm (27.10). 

These observations allow us to modify Algorithm (27.10) by managing 
the storage in a working array G that needs only n t +l columns. Specifically, 
let G denote a ft X («* + 1) array of storage with n satisfying Eq. (27.9). 
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The working array G will be partitioned by the algorithm into three sub¬ 
arrays </„ G lt and G t as follows: 

(27.15) G, = rows 1 through /, — 1 of G 

(27.16) G t — rows /, through i, — 1 of G 

(27.17) G t = rows /, through /, + m, — 1 of G 

The integers /, and /, are determined by the algorithm and their values change 

in the course of the processing. These integers are limited by the inequalities 

At the various stages of the algorithm the (it* 4- l)st column of the G 
array contains either the vector 6, [e.g., see the left side of Eq. (27.4)] or the 
processed vector d, [e.g., see the right side of Eq. (27.5)]. 

For 1 <.j<.n the identification of matrix elements with storage locations 
is as follows: 

(27.18) In G t : storage location (ij) contains matrix element 

1) 

(27.19) In Gt: storage location (t,j) contains matrix element 
(«,/,+;-1) 

(27.20) In G ,: storage location (ij) contains matrix element 

VJ'+J- 1) 

To indicate the primary idea of a compact storage algorithm for the 
banded least squares problem consider Fig. 27.1 and 27.2. If Algorithm (27.10) 
were applied to the block diagonal problem with n* — 4, then at the step in 
which the data block [C„ h,] is introduced the working array might appear as 
in Fig. 27.1. Taking advantage of the limited bandwidth these data can be 
packed in storage as in Fig. 27.2. 

In this illustration the inequalities i, < j, <, i, are satisfied. For further 
diagrammatic illustration of this algorithm the reader may wish to look 
ahead to Section 3 and Fig. 27.4 and 27.5. 

A detailed statement of the algorithm follows. In describing the algorithm 
we shall use the notation G(i,j) to denote the (i,j) element of the array G 
and the notation G{iy : /*,/,: j t ) to denote the subarray of G consisting of 
all elements G(i,j) with and j t <Zj<Zj 2 . 

(27.21) Algorithm BSEQHT ( Banded Sequential Householder Trian- 
gularization ) 

Step Description 

1 Set/, := 1 and/, := 1. 

2 For tl,... ,q,do Steps 3-24. 
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Fig. 27.1 Introducing the data C, and b,. 


Step Description 

3 Remarks: At this point the data [C,: b,] and the integers 
m, and j, must be made available to the algorithm. It is as¬ 
sumed that m, > 0 for all t and/, ^^ • • ♦ "^Jt ^ 1. 

4 Set G(i,: i, + m, — 1,1: n* + 1) := [C,: b,] [See Eq. (27.14) 
for a definition of C,. Note that the portion of the G array 
into which data are being inserted is the subarray G s of 
Eq. (27.17).] 

5 If j, = /,, go to Step 18. 

6 Remark: Here the monotonicity assumption on j, assures 
thaty, > 

Ify, <i i„ go to Step 12. 


7 
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> SUBARRAY G 


1 


J 

'v 


SUBARRAY G. 



SUBARRAY G. 


Fig. 27 J The same data as in Fig. 27.1 but packed into the storage 

array, G. 

Step Description 

8 Remark: Here j, exceeds This indicates rank deficiency 
since the diagonal element in position (/„ /,) of the trian¬ 
gular matrix R [see Eq. (27.6)] will be zero. Nevertheless, 
the triangularization can be completed. Some methods for 
solving this rank-deficient least squares problem will be 
discussed in the text following this algorithm. 

9 Move the contents of G(i r : i, -f m, — 1,1: n t + 1) into 

+ 1,1 :n, + 1). 

10 Zero the subarray <?(/,: — 1,1 :n t + 1). 

11 Set i r :—j t . 

12 Set p :== min (n* — 1— 1); if p — 0, go to Step 
17. 

13 For / := 1. p, do Steps 14-16. 

14 Set k := min (l,j, — /,). 
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Step Description 

15 For I := / + 1,..., »*, set G(i p 4- /, i — k) := GQ, 4- /, 
/)• 

16 For f := 1,..., k, set G(i, + /,»*+ 1 — i) := 0. 

17 Set i p := Jr [Note that this redefines the partition line 

between subanrays G, and G a of Eq. (27.15) and (27.16).] 

18 Remark: Steps 19 and 20 apply Householder triangulariza- 
tion to rows i, through i, + m, — 1. As was discussed 
following the statement of Algorithm SEQHT (27.10), 
some reduction of execution time may be attainable by re¬ 
placing the general Householder method by one of the 
methods based on 2 x 2 rotations or reflections. 

19 Set rfi := /, + m, — i r Set fc := min (n h + 1, /ft). 

20 For / :=* 1,..., £, execute Algorithm HI (/, max (i 4- 1, 

i r ~i,+ 1), *>, G(i„ i), p, G(l r , / 4- D, »* 4- 1 - i). 

21 Set i r i, + 1c. [Note that this redefines the partition line 
between subarrays G a and G, of Eq. (27.16) and (27.17).] 

22 If £ < a* + 1, go to Step 24. 

23 For j 1. n p , set G(i t — 1 ,j) :== 0. 

24 Continue. 

25 Remark: The main loop is finished. The triangular matrix 
of Eq. (27.6) is stored in the subarrays G, and G 2 [see Eq. 
(27.15) and (27.16)] according to the storage mapping of 
Eq. (27.18) and (27.19). 

If the main diagonal elements of the matrix R of Eq. 
(27.6) are all nonzero, the solution of problem (27.7) can 
be computed by back substitution using Steps 26-31. Note 
that these diagonal elements of the matrix A are used as 
divisors at Step 31. Discussion of some techniques for 
handling the alternative (singular) case are given in the 
text following this algorithm. 

26 For i := 1,..., n, set X{i) := G(/, n„ + 1). 

27 For i := n, n — 1,..., 1 do Steps 28-31. 

28 Set s := 0 and / := max (0, / — /„). 

29 If i = n, go to Step 31. 

30 For j 2,..., min (n + 1 — i, n t ), set 5 1=5 + 
G(U + l) x X(i-l +j). 

31 Set X(i): = [JSf(/) - a]/G(f, l + 1). 
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Step Description 

32 Remark: The solution vector x is now stored in the array 
X. If, as would usually be the case, the full data matrix 
[A: b ] [see Eq. (27.2) and (27.3)] has more than n rows, 
the scalar quantity, e, of Eq. (27.6) will be stored in location 
G(n + 1,»» 4-1). The magnitude of e is the norm cf the 
residual associated with the solution vector, x. 

The banded structure of A does not generally imply that the unsealed 
covariance matrix C of Eq. (12.1) will have any band-limited structure. How¬ 
ever, its columns, c )t j— 1. n, can be computed, one at a time, without 

requiring any additional storage. Specifically the vector c, can be computed 
by solving the two banded triangular systems 

(27.22) R T w, = e, 

(27.23) Rcj = wj 

where e, is the /th column of the n x n identity matrix. 

The computational steps necessary to solve Eq. (27.22) or more generally 
the problem KFy =z for arbitrary z are given in Algorithm (27.24). Here it is 
assumed that the matrix R occupies the array G in the storage arrangement 
resulting from the execution of Algorithm (27.21). The right-side vector, for 
example e, for Eq. (27.22), must be placed in the array X. Execution of Al¬ 
gorithm (27.24) replaces the contents of the array X by the solution vector, 
for example, w, of Eq. (27.22). 

(27.24) Algorithm Solving RFy — z 

Step Description 

1 For j := 1.n, do Steps 2-6. 

2 Set s 0. 

3 If / = 1, go to Step 6. 

4 Set /, := max (1 J — n t + 1). Set := y — 1. 

5 For i :=* i t , set s s + X(i) x G(i,j — i + 1 + 

max (0, i — /,)). 

6 Set X(J) := (X(J) - s)/G(j, 1 + max (0 J - /,)). 

To solve Eq. (27.23) for c, after solving Eq. (27.22) for w, one can simply 
execute Steps 27-31 of Algorithm (27.21). 

Furthermore, the quantity a 1 of Eq. (12.2) may be computed as 
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where e is defined in Eq. (27.fi). The quantity e is computed by the Algorithm 
BSEQHT as noted at Step 32. 

Fortran subroutines BNDACC and BNDSOL are included in Appendix 
C as an implementation of Algorithm BSEQHT (27.21) and Algorithm 
(27.24). Use of these subroutines is illustrated by the Fortran program 
PROGS of Appendix C, which fits a cubic spline curve to discrete data and 
computes the covariance matrix of the coefficients using the technique of 
Eq. (27.22) and (27.23). 

The remarks at the end of Section 1 regarding possible implementations 
of Algorithm SEQHT (27.10) for sequential estimation apply with some 
obvious modification to Algorithm BSEQHT also. The significant difference 
is the fact that in Algorithm BSEQHT the number of columns having nonzero 
entries is typically increasing as additional blocks of data are introduced. 
Thus the number of components of the solution vector that can be determined 
is typically less at earlier stages of the process than it is later. 

At Steps 8 and 25 mention was made of the possibility that the problem 
might be rank-deficient. We have found that the Levenberg idea (see Chapter 
25, Section 4) provides a very satisfactory approach to this situation without 
increasing the bandwidth of the problem. We refer to the notation of Eq. 
(25.30) to (25.34). 

Assume the matrix X of Eq. (25.31) is banded with bandwidth n*. For 
our present consideration assume that F is also banded with bandwidth not 
exceeding n». It is common in practice for the F matrix to be diagonal. By 
appropriate row interchanges the matrix 



of Eq. (25.31) can be converted to a new matrix 

[i:$] 

in which the matrix A has bandwidth n h . The matrix [A: $] can then be pro¬ 
cessed by Algorithm BSEQHT. If the matrix F is nonsingular, then A is of 
full rank. If the matrix F is nonsingular and A is sufficiently large, then A will 
be of full pseudorank so that Algorithm BSEQHT can be completed including 
the computation of a solution. 

If one wishes to investigate the effect of different values of A in problem 
(25.31), one could first process only the data [X: B\ of problem (25.31), re¬ 
ducing this matrix to a banded triangular array 



Assume that this matrix T is saved, possibly in the computer’s secondary 
storage if necessary. 
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Then to solve problem (2S.31) for a particular value of X, note that an 
equivalent problem is 


R 


s 

0 


e 

_XF 


M. 


The latter problem can be subjected to row interchanges so that the coef¬ 
ficient matrix has bandwidth n t , and this transformed problem is then in the 
form such that its solution can be computed using Algorithm BSEQHT. 

This technique of interleaving rows of the matrix [AF: Xd\ with the rows 
of the previously computed triangular matrix [A: d\, to preserve a limited 
bandwidth, can also be used as a method for introducing new data equations 
into a previously solved problem. This avoids the need for triangularization 
using the entire original data set for the expanded problem. 

In the case where F = /, and d— 0, the choice of parameter X can be 
facilitated by computing the singular values and transformed right side of the 
least squares problem of Eq. (27.7). Thus with a singular value decomposition 

R = USV r 

the vector 

(27.25) g = U T i 

and the matrix 

S = Diag {ai,.... a,) 

can be computed without additional arrays of storage other than the array 
that contains the banded matrix R. The essential two ideas are: post- and 
premultiply R by a finite sequence of Givens rotation matrices J, and T, so 
that the matrix 21 = T, • • • TiRJ, • • • /, is bidiagonal. The product T, • • • T,d 
replaces d in storage. Compute the SVD of the matrix B using Algorithm 
QRBD (18.31). The premultiplying rotations are applied to <? in storage, 
ultimately producing the vector g of Eq. (27.25). A value of X can then be 
determined to satisfy a prespecified residual norm or solution vector norm by 
using Eq. (25.48) or (25.47), respectively. 

Section 3. AN EXAMPLE: UNE SPLINES 

To fix some of the ideas that were presented in Sections 1 and 2, consider 
the following data-fitting (or data-compression) problem. Suppose that we 
have m data pairs {(/„ y,)} whose abscissas occur in an interval of t such that 
a i = 1,..., m. It is desired that these data be fit by a function 

f(t) whose representation reduces the storage needed to represent the data. 
Probably the simplest continuous function (of some generality) that can be 
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(least squares) fit to these data is the piecewise linear continuous function 
defined as follows: 

(27.26) Partition the interval [a, 6] into n — 1 subintervals with breakpoints 
t w satisfying a = t ,u < t <v < • • • < t M — b. 


For 

(27.27) 
define 

(27.28) h*) = 

(27.29) zit) - 1 - wX 0 - 
and 

(27.30) /(f) = x,*Xf) 4- *,+, wXO /= 11 

The n parameters x„ i — I,..., n, of Eq. (27.29) are to be determined as 
variables of a linear least squares problem. 

As an example, let us take m — 10 and n — 4 with breakpoints evenly 
spaced. A schematic graph for such a problem is given in Fig. 27.3. 



t 0> t (2) f (3> t (4) 


Fig. 27.3 A schematic example of a line spline with m ■* 10, n ** 
4, and data points (/„ yd indicated by dots. 
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The least squares problem for the x, then has the form shown in Fig. 27.4. 
Note that the coefficient matrix is banded with bandwidth n» = 2. 
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Fig. 27.4 


The progress of the band-limited algorithm BSEQHT (27.21) for this 
example is diagramed in Fig. 27.5. 
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Stage (1) Initially i p = i r — 1. The first block of data, consisting of the 
nontrivial data in the first three rows of Fig. 27.4, is introduced. Set = 1, 
m, = 3. 

Stage (2) This block is triangularized using Householder transforma¬ 
tions. Set i, — 4. 

Stage (3) Introduce second block of data from Fig. 27.4. Set j % — 2, 
m t = 3. 

Stage (4) Left shift of second row exclusive of the last column, which 
represents the right-side vector. Set i, = j t (== 2). 

Stage (S) Triangularization of rows 2-6. Set i, = 5. 

Stage (6) Introduce third block of data from Fig. 27.4. Set j 3 = 3, 
m } = 4. 

Stage (7) Left shift of row 3 exclusive of last column. Set i„ = j 3 (= 3). 

Stage (8) Triangularization of rows 3-8. Set i, = 6. 

The data appearing at stage (8) represent the least squares problem shown 
in diagram (9). This problem can now be solved by back substitution. 

As an illustration of the extent of storage that can be saved with the use 
of this band-matrix processing of Section 2, consider an example of line- 
spline fitting with m — 1000 data points using 100 intervals. Thus the least 
squares problem will have n = 101 unknowns. Let us suppose further that 
the row dimension of each block we shall process does not exceed 10. Then 
the maximum size of the working array does not have to exceed [n + 1 + 
max (m,)] x 3 — (101 + 1 + 10) x 3 = 336. The use of a less specialized 
sequential accumulation algorithm such as that of Section 1 of this chapter 
would require a working array of dimension at least [n + 1 + max (m r )] x 
(« -f 1) = (101 + 1 4- 10) x 102 = 11,424. If all rows were brought in at 
once, the working array would have to have dimensions m x (n + 1) = 1000 
X 101 = 101,000. 


Section 4. DATA FITTING USING CUBIC 
SPLINE FUNCTIONS 

As a further example of the use of sequential processing for banded least 
squares problems we shall discuss the problem of data fitting using cubic 
spline functions with uniformly spaced breakpoints. The Fortran program 
PROGS given in Appendix C implements the approach described in this 
section. 

Let numbers h, < b t < ■ • • < b„ be given. Let S denote the set of all 
cubic spline functions defined on the interval [hi, b n ] and having internal 
breakpoints (often called knots) b t ,..., h„_,. A function/is a member of S 
if and only if /is a cubic polynomial on each subinterval [b k , h* +1 ] and is 
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continuous together with its first and second derivatives throughout the 
interval \b u hj. 

It can be shown that the set S constitutes an (n + 2)-dimensional linear 
space of functions. Thus any set of n + 2 linearly independent members of 
S, say {qj: j = 1,...,» + 2], is a basis for S. This implies that each/ e S 
has a unique representation in the form 

/(*) = jg Cjqfx) 

Using such a basis, the problem of finding a member of S that best fits a 
set of data [{x„ y,) :x, e [b lt 6J; i = 1,..., m) in a least squares sense takes 
the form 


where 



y 


a„ = qfx,) 

Here c is the (n + 2)-vector with components c, and y is the m-vector with 
components y,. 

There exist bases for S with the property that if the data are ordered so 
that Xi <; x t <, • • • <; x m , the matrix A will be band-limited with a band¬ 
width of four. Methods for computing such basis functions for unequally 
spaced breakpoint sets are described in Carasso and Laurent (1968), de Boor 
(1971 and 1972), and Cox (1971). Here we restrict the discussion to uniformly 
spaced breakpoint sets in order to avoid complexities not essential to the 
illustration of banded sequential processing. 

To define such a basis for a uniformly spaced breakpoint set let 

h = 6 *+1 Ar=l,...,n 1 

Define the two cubic polynomials 

/>,(/) - 0.25/ 3 
and 

p a (r) = 1 - 0.75(1 + /XI - 0 1 

Let /, denote the closed interval [6,. b 2 ] and let /* denote the half-open in¬ 
terval ( b k , 6*+,] for k = 2,.... n — 1. 

In the interval I k only four of the functions qj have nonzero values. These 
four functions are defined for x e /* by 
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9*(*) “ PlO - 0 

= Pi(\ ~ 0 
9k+i(x) = Pi(t) 

$*+»(*) = PiO) 

The Fortran program PROQ6 in Appendix C solves a sample data-fitting 
problem using this set of basis functions and sequential Householder accu¬ 
mulation of the resulting band-limited matrix. The data for this curve fitting 
example are given in Table 27.1. 


Table 27.1 data for cubic spline-foting example 


X 

y 

X 

y 

2 

2.2 

14 

3.8 

4 

4.0 

16 

5.1 

6 

5.0 

18 

6.1 

8 

4.6 

20 

6.3 

10 

2.8 

22 

5.0 

12 

2.7 

24 

2.0 


A parameter NBP is set to the values 5, 6,7,8,9, and 10, respectively, to 
specify the number of breakpoints for each of the six cases. In terms of the 
parameter NBP the breakpoints for a case are defined as 

bt = 2 + i = l .NBP 

The number of coefficients to be determined is NC = NBP + 2. Note 
that in the sixth case the number of coefficients is equal to the number of data 
points. In this sixth case the fitted curve interpolates the data. 

Define 

RMS = 



where r, is the residual at the Ah data point. The value of RMS for each case 
is given in Table 27.2. Note that RMS is not a monotone function of NBP. 

Table 27.2 RMS AS a function of NBP 


NBP 

5 

6 

7 

8 

9 

10 

RMS 

0.254 

0.085 

0.134 

0.091 

0.007 

0.0 


Plots of each of the six curve fits are given in Fig. 27.6. 
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(27.31) C = [A : b] 

Suppose C has m rows and n columns and is of rank k. Suppose further that 
a Cholesky factor R for C has been computed. Here A is a & x n upper trian¬ 
gular matrix satisfying 

(27.32) QC = [*] 

for some m x m orthogonal matrix Q. The matrix R also satisfies 

(27.33) R r R = C T C 

It will be convenient to make the further assumption that the k diagonal 
elements of R are all nonzero. This will necessarily be true if k = n. If k < it, 
this can always be achieved by appropriate column interchanges and re- 
triangularization if necessary. 

Let if denote a row of C that is to be removed. Without loss of generality 
we shall assume that tF is the last row of C. Let C denote the submatrix con¬ 
sisting of the remaining rows of C. 

(27.34) C - 

Note that the rank of € may be either k or k — 1. The practical applications 
of data removal most commonly involve only the case in which Rank (C) = 
Rank (C) = it; however, we shall consider the general case of Rank (C) — 1 
<; Rank_(£) <. Rank (C) — k<,n as long as it does not require a signifi¬ 
cant amount of additional discussion. 

We wish to find a Cholesky factor R for C. Thus R will be an upper trian¬ 
gular k x it matrix having the same rank as C and satisfying 

(27.35) QC = ^ J 

for some (m — 1) x (m — 1) orthogonal matrix Q. Such a matrix R will also 
satisfy 

(27.36) R r R = C T C = C T C - vif 

= R T R — wF 

The first two methods to be described appear in Gill, Golub, Murray, and 
Saunders (1972). 
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Row Removal Method 1 

In this method it is assumed that the matrix Q of Eq. (27.32) is available 
as well as the matrix R. Partition Q and rewrite Eq. (27.32) as 


*{ 

Qi P 

rCi 

R 

(27.37) 1{ 

u T a 

•Lr]“ 

0 

m-k-\{ 

.Qt 9. 


.0. 


W* W-» 


ffl -1 1 

Left-multiply both sides of Eq. (27.37) by an orthogonal matrix that trans¬ 
forms the vector q to zero, modifies a, and leaves the first k rows of Q un¬ 
changed. This can be accomplished, for instance, by a single Householder 
transformation or by (m — k — 1) Givens transformations. Equation (27.37) 
is thereby transformed to 



'Qt P 

rCi 

R' 

(27.38) 

a r & 
A o. 

■[J* 

0 

.0. 


Next left-multiply both members of Eq. (27.38) by a sequence of k Givens 
transformations that successively transform each component of p to zero 
while modifying fi. Let G„ denote a Givens transformation matrix operating 
on rows i and j. First left-multiply by G M+1 , next by and so forth 

with the final transformation matrix being G,* +l . This sequence of operations 
assures that R is transformed to an upper triangular matrix, say, R. 

By these transformations Eq. (27.38) is transformed to 



o 


R' 

(27.39) 

& s 

A o. 

*[« r ] = 

& 

.0. 


Since £ is the only nonzero element in its column and the column is of unit 
euclidean length it follows that £ — ±1. Furthermore since the rows also 
have unit euclidean length u must be zero. This in turn implies that w T — £v r 
= ±» r . Thus Eq. (27.39) can be written as 



Qi 

O' 

r Ci 

' R ' 

(27.40) 

0 

±i 

•[3- 



A 

0. 


. 0 . 


Defining 

< 2M1 > s ~ [&] 
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it follows that Q is an (m — 1) x (m — 1) orthogonal matrix, & is a k x » 
upper triangular matrix and 

(27.42) Qt - ] 

Thus Q and R satisfy the conditions required of Q and R in Eq. (27.35) and 
(27.36). 

The question of whether the rank of R is k or k — 1 depends upon whether 
A in Eq. (27.38) is nonzero or zero. By assumption the Jc diagonal elements 
of R are each nonzero. If A is nonzero, it is easily verified by noting the 
structure and effect of the Givens transformation matrices used in passing 
from Eq. (27.38) to Eq. (27.40) that the k diagonal elements of R will also be 
nonzero. 

Suppose on the other hand that A is zero. Then ||p|| = 1, which assures 
that some components of p are nonzero. Let p, denote the last nonzero com* 
ponent of p, i.e., p, 9 * 0, and if / 9 * k, then p, = 0 for / < / k. In the order 
in which the matrices G, t are applied in transforming Eq. (27.38) to Eq. 
(27.40), the matrix'G ;i * +1 will be the first transformation matrix that is not 
simply an identity matrix. This matrix G, k+l will be a (signed) permutation 
matrix that interchanges rows / and k + 1 , possibly changing the sign of one 
of these rows. In particular, its effect on R will be to replace row / with a row 
of zero elements. Subsequent transformations will not alter this row. There¬ 
fore, row / of R in Eq. (27.40) will be zero. 

Thus Rank (R) will be less than k. The rank of R cannot be less than k—\ 
since Rank (R) — Rank (£) ;> Rank (C) — 1 = k — 1. 

Row Removal Method 2 

In a problem in which m is very large or in which data are being accumu¬ 
lated sequentially one would probably not retain the matrix Q of Eq. (27.32). 
In this case Method 1 cannot be used directly. Note, however, that the only 
information from the matrix Q that contributes to the determination of R in 
Method 1 is the vector p of Eq. (27.37) and (27.38) and the number A of Eq. 
(27.38). We shall see that these quantities p and A can be computed from R 
and v. 

Rewrite Eq. (27.32) as 

(27.43) C = (?•[* ] 

Then using the partitions defined in Eq. (27.37) it follows that 

(27.44) v r = fR 
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or equivalently 

(27.45) R T p = v 

If Mb of rank n, Eq. (27.45) constitutes a nonsingular triangular system that 
can be solved for p. 

If the rank k of R is less than n, the system in Eq. (27.45) is still consistent 
and has a unique k-dimensional solution vector p. By assumption the first k 
rows of JT constitute a k x k triangular submatrix with nonzero diagonal 
elements. This submatrix along with the first k components of v defines a 
system of equations that can be directly solved for p. 

Having determined p the number A can be computed as 

(27.46) A - (1 - |!p IP)" 1 

as a consequence of the unit euclidean length of the column vector (p r . A, 0 if 
in Eq. (27.38). The sign of A is arbitrary and so can be taken to be nonnega¬ 
tive, as is implied by Eq. (27.46). 

Having computed p and A one can compute R by using k Givens transfor¬ 
mations as described following Eq. (27.38) in Method 1. This part of the com¬ 
putation is expressed by the equation 

(27.47) G ’.*+.Gm+« o] = [±1 ±*] 

It is to be expected that Method 2 will be a somewhat less accurate com¬ 
putational procedure than Method 1 because of the necessity of computing p 
and A from Eq. (27.45) and (27.46). The accuracy with which p is determined 
will be affected by the condition number of R, which, of course, is the same 
as the condition number of C. 

This accuracy limitation appears to be inherent, however, in any method 
of updating the Cholesky factor R to reflect data removal that does not make 
use of additional stored matrices, such as Q or C. For a given precision of 
computer arithmetic, updating methods that operate directly on the matrix 
(A T A)~ l may be expected to be significantly less reliable than Method 2. 

Row Removal Method 3 

The third method to be described is of interest because it requires fewer 
operations than Method 2 and because it is algorithmically very similar to 
one of the methods for adding data. This permits the design of fairly compact 
code that can handle either addition or removal of data. The comparative 
numerical reliability of Methods 2 and 3 has not been studied. 
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This method has been discussed by Golub (1969), Chambers (1971), and 
Gentleman (1972a and 1972b). Following the practice of these authors we 
shall restrict our discussion to the case in which Rank (€) — Rank (C) = k 
= n. Properties of the more general case will be developed in the Exercises. 

Letting i denote the imaginary unit (I 1 — —1), Eq. (27.36) can be written 
as 


(27.48) 




By formal use of the Givens equations (3.S), (3.8), and (3.9), matrices 
F m can be defined that will triangularize as follows: 


(27.49) 

(27.50) 

(27.51) 



Here the matrix F* differs from the identity matrix only at the intersec¬ 
tions of rows / and k + 1 with columns / and k + 1. In these four positions 
F* contains the submatrix 


(27.52) 

r <r* h a n 

L—/r* a*J 

where 


(27.53) 

<5* = 

(27.54) 

p“> == 

(27.55) 


(27.56) 

- - V 


It can be shown that our assumption that both C and C are of full rank n 
implies that the numbers t5* defined by Eq. (27.53) are positive. 

The multiplication by F* indicated in Eq. (27.50) can be expressed entirely 
in real arithmetic as follows: 
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(27.57) rf) = o a 'r?r» - T m v'j~" 1 

(27.58) vf = -t »/#-» + j ^ ~. * 

The method of defining F w assures that r# 1 = p <w and = 0. Fur¬ 
thermore, it can be easily verified that v ik) = 0, that each matrix It® is upper 
triangular, that F wtr F m = /, and that 


(27.59) 



Thus it satisfies the conditions required of it; i.e., R is an upper triangular 
Cholesky factor for the matrix C of Eq. (27.34). 

In practice the inherent potential instability of the data-removal problem 
will be reflected in loss of digits in the subtraction by which 3 (t> is computed 
and in the fact that the multipliers o m and t® in Eq. (27.57) and (27.58) can 
be large. 

Gentleman (1972a and 1972b) points out that modifications of the Givens 
method that maintain a separate array of squared row scale factors [see Eq. 
(10.32)] are particularly adaptable to Row Removal Method 3. Simply by 
permitting a squared row scale factor to be negative one can represent a row 
such as iif of Eq. (27.48) that has an imaginary scale factor. Thus by per¬ 
mitting negative as well as positive values of d 2 in Eq. (10.31) and d 2 in 
Eq. (10.33) the method given in Eq. (10.31) to (10.44) will handle data 
removal as well as data augmentation. Note that when d 2 is negative the 
quantity t of Eq. (10.35) is no longer bounded in the interval [0,1]; hence the 
problem of avoiding overflow or underflow of £> and 3 becomes more 
involved. 


EXERCI8ES 

(27.60) [Chambers (1971)] 

(a) Show that the numbers a u> and t® of Eq. (27.55) and (27.56) 
can be interpreted as being the secant and tangent, respectively, 
of an angle 6 m . 

(b) Show that these angles 6 ,n are the same angles that appear in 
the Givens transformations G m (c a> — cos 6 a> , s (l) — sin $ a> ) 
which would occur in the process of adjoining the row if to R 
to produce R, i.e., in the operations represented by 

«”••• «"*[£]-[ 2 ] 
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(c) With 0 a \ c"\ and s a> defined as above, show that as an alterna¬ 
tive to Eq. (27.58) vf could be computed as 

vf = —s a> rff + c u, t>y-‘> 


where c a> = l/a a) and s a) => x m la m . 

(27.61) Show that Row Removal Method 3 can theoretically be extended to 
the general case of Rank (C) — 1 <; Rank (£) ^ Rank (C)si^ 
n using the following observations. (Continue to assume, as at the 
beginning of Section 5, that the k diagonal elements of R are non¬ 
zero.) 

(a) If d (<> is nonpositive for some / = 1. k, let h be the index 

of the first such value of 8 m , i.e., i <u <; 0 and if h & 1 , 3 a) > 0 
for 1 <, l < h. Show that 3 IU — 0. 

(b) With h as above it follows that = erjjr ,} where e =* +1 or 
e = —1. In this case show that it must also be true that vf~ u 
— fory = h + 1, ..., n. 

(c) With h as above, show that it is valid to terminate the algorithm 
by defining the final matrix R to be the matrix with row 
h set to zero. 


APPENDIX 


BASIC LINEAR ALGEBRA 
INCLUDING PROJECTIONS 



In this appendix we list the essential facts of linear algebra that are used 
in this book. No effort is made to present a logically complete set of con¬ 
cepts. Our intention is to introduce briefly just those concepts that are directly 
connected with the development of the book’s material. 

For a real number x define 


sgn (x) «= 



if x^>0 
if x < 0 


An n -vector x is an ordered n-tuple of (real) numbers, x,.x„. 

An m x it matrix A is a rectangular array of (real) numbers having m 
rows and it columns. The element at the intersection of row i and column j is 
identified as a„. We shall often denote this m x n matrix by A mxm . 

The transpose of an m x n matrix, A, denoted A T , is the it x m matrix 
whose component at the intersection of row i and column / is a Jt . 

The matrix product of an m x n matrix A and an / x k matrix B, written 
AB, is defined only if / = it, in which case C = AB is them x k matrix with 
components 

n 

Cti — Onbi, + a n bu + • • • + o,„b nJ — £ a tl b pj 

/-i 

Frequently it is convenient to regard an n-vector as an it x 1 matrix. In par¬ 
ticular, the product of a matrix and a vector can be defined in this way. 

It is also often convenient to regard an m x n matrix as consisting of 
m n-vectors, as its rows, or it m-vectors as its columns. 

The inner product (also called scalar product or dot product) of two n- 
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dimensional vectors u and v is defined by 

s = u T v = ^ u, v, 

Two vectors are orthogonal to each other if their inner product is zero. 

The euclidean length or euclidean norm or l x norm of a vector v, denoted 
by ||t>||, is defined as 

This norm satisfies the triangle inequality, 

ll» + »ll<:il««ll + ll*ll 


positive homogeneity. 


where a is a number and u is a vector, and positive definiteness, 

|| u ||>0 ifi /^0 
|ju|! = 0 if a = 0 

These three properties characterize the abstract concept of a norm. 

The spectral norm of a matrix A, denoted by ||/4||, is defined as 

(A.l) IMII = max {||i4»j|:||®|| = 1} 

We shall also have occasion to use the Frobenious norm (also called the 
Schur or euclidean matrix norm) of a matrix A, denoted by IMII, and defined 
as 

(A.2) = 

The spectral and Frobenious norms satisfy 

(A.3) max,, | a„ | £ \\A || <£ \\A ||, ^ **'* )\A || 

where A is an m x n matrix and k — min (m, n). 

Both of these matrix norms satisfy the three properties of an abstract 
norm plus the multiplicative inequalities 


IMBIKIMIMIBII 

lM*|l,£lM[HI*llr 
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We shall let the numeral 0 denote the zero vector or the zero matrix with 
the distinction and dimension to be determined by the context. 

A set of vectors v . . is linearly dependent if there exist scalars a,, 

not all zero, such that 

(A.4) jfj a,v, — 0 

Conversely if condition (A.4) holds only with a, = • • • = a* = 0, the vec¬ 
tors are linearly independent. 

The set of all it-dimensional vectors is an n -dimensional vector space. Note 
that if u and v are members of this vector space, then so is u + v and av 
where a is any scalar. These two conditions of closure under vector addition 
and scalar-vector multiplication characterize the abstract definition of a 
vector space; however, we shall be concerned exclusively with the particular 
finite-dimensional vector space whose members are n-tuples of real numbers. 

If a subset Tof a vector space S is closed under vector addition and scalar- 
vector multiplication, then 7* is called a subspace. There is a maximal number 
of vectors in a subspace T that can be linearly independent. This number, m, 
is the dimension of the subspace T. A maximal set of linearly independent 
vectors in a subspace I* is a basis for T. Every subspace To f dimension m i> 1 
has a basis. In fact, given any set of k < m linearly independent vectors in an 
m-dimensional subspace T, there exist m — k additional vectors in T such 
that these m vectors together constitute a basis for T. If the vectors u„ ..., 
u m constitute a basis for Tand v e T, then there exists a unique set of scalars 
a, such that v — 2,1, a,u,. 

The span of a set of vectors v . . is the set of all linear combina¬ 

tions of these vectors, i.e., the set of all vectors of the form u — £,* , a,v, for 
arbitrary scalars a,. The span of a set of k vectors is a subspace of dimension 
m k. 

There are certain subspaces that arise naturally in connection with ma¬ 
trices. Thus with an m X n matrix A we associate the range or column space, 
which is the span of its column vectors; the null space or kernel, which is the 
set {x: Ax — 0}; and the row space, which is the span of the row vectors. 
Note that the row space of A is the range of A T . 

It is often convenient to note that the range of a product matrix, say, A — 
I JVW, is a subspace of the range of the leftmost factor in the product, here 
U. Similarly the row space of A is a subspace of the row space of the right¬ 
most factor, here W. 

The row and column spaces of a matrix A have the same dimension. This 
number is called the rank of the matrix, denoted by Rank (A). A matrix A nx „ 
is rank deficient if Rank (A) < min (m, n) and of full rank if Rank (A) = min 
(m, n). A square matrix A mx , is nonsingular if Rank (A) — n and singular if 
Rank (A) < n. 
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A vector v is orthogonal to a subspace T if v is orthogonal to every vector 
in T. It suffices for v to be orthogonal to every vector in a basis of T. A sub¬ 
space T is orthogonal to a subspace U if t is orthogonal to u for every t e T 
and u e U. If two subspaces T and U are orthogonal, then the direct sum of 
T and U, written V = T® U, is the subspace consisting of all vectors {w: v 
= t + u,t e T,u e U }. The dimension of V is the sum of the dimensions 
of T and U. 

If T and U are mutually orthogonal subspaces of an a-dimensional vector 
space S and S = T® U, then Tand U are called orthogonal complements of 
each other. This is written as T = U x and U — T 1 . For any subspace T, T 1 
exists. If T is a subspace of S and s € S, then there exist unique vectors t e 
T and u e T x such that $ = ! + «. For such vectors one has the Pythagorean 
condition || j ||* = ||/|| 2 + || ii ||*. 

A linear fiat is a translated subspace; i.e., if T is a subspace of S and s € 
S, then the set L — {»: v — s 4- /, t e T } is a linear flat. The dimension of a 
linear flat is the dimension of the unique associated subspace T. A hyperplane 
H in an n-dimensional vector space S is an (n — l)-dimensional linear flat. 

If H is a hyperplane and h 0 e H, then T = {t: t ~ h — h 0 , h 6 H] is an 
(n — l)-dimensional subspace and T x is one-dimensional. If u e T x , then 
u T h takes the same value, say, d, for all h 6 H. Thus given the vector u and 
the scalar d, the hyperplane H can be characterized as the set {x: u T x — d). It 
is from this type of characterization that hyperplanes arise in practical com¬ 
putation. 

A halfspace is the set of vectors lying on one side of a hyperplane, i.e., a 
set of the form {x: u T x ;> d}. A polyhedron is the intersection of a finite num¬ 
ber of halfspaces, i.e., a set of the form {x: ujx ;><?,,/— 1 ,... ,m}or equi¬ 
valently {x: Ux > d) where U is an m x n matrix with rows uf, d is an m 
vector with components d„ and the inequality is interpreted termwise. 

The elements a„ of a matrix A mx „ are called the diagonal elements of A. 
This set of elements, {a„}, is called the main diagonal of A. If all other elements 
of A are zero, A is a diagonal matrix. If a„ — 0 for | i — j | > 1, A is tridiago¬ 
nal. 

If a tJ — 0 for j < i and j > i + 1, A is upper bidiagonal. If a tJ — 0 for 
j < i, A is upper triangular. If a„ = 0 for j < i — l, A is upper Hessenberg. 
If A T is upper bidiagonal, triangular, or Hessenberg, then A is lower bidi¬ 
agonal, triangular, or Hessenberg, respectively. 

A square n x n diagonal matrix with all diagonal elements equal to unity 
is an identity matrix and is denoted by /, or by I. 

If BA — /, then B is a left inverse of A. A matrix A m%n has a left inverse if 
and only if Rank (A) — n <. m. The left inverse is unique if and only if 
Rank (A) = n = m. By transposition one similarly defines a right inverse. 

If A is square and nonsingular, there exists a unique matrix, denoted A~', 
that is the unique left inverse and the unique right inverse of A and is called 
the inverse of A. 
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A generalization of the concept of an inverse matrix is the pseudoinverse, 
denoted A*, that is uniquely defined for any matrix A mxn and agrees with the 
inverse when A is square nonsingular. The row and column spaces of A + 
are the same as those of A T . The pseudoinverse is intimately related to the 
least squares problem and is defined and discussed in Chapter 7. 

A square matrix Q is orthogonal if Q T Q = /. It follows from the unique* 
ness of the inverse matrix that QQP — / also. 

A set of vectors is orthonormal if the vectors are mutually orthogonal and 
of unit euclidean length. Clearly the set of column vectors of an orthogonal 
matrix is orthonormal and so is the set of row vectors. 

If Qmx*, m ;> n, has orthonormal columns, then || Q || = 1,|I2IU = n m , 
II Q A II = Mil.and|| QA || r = MU,. If m<,n, has orthonormal rows, 
then ||0|| = 1, || fill, = m l/i , \\AQ\\ = Mil, and \\AQ\\ r = |M||,. Note 
that if Q nxn is orthogonal, it satisfies both these sets of conditions. 

Some particular orthogonal matrices that are computationally useful are 
the Givens rotation matrix, 

T cos# sm0~l 
L~sin0 cos 

the Givens reflection matrix, 

[“cos 0 sin 01 
|_sin 0 —cos 0\ 

and the Householder reflection matrix, 



1-2 


uu T 


for an arbitrary nonzero vector u. 

A permutation matrix is a square matrix whose columns are some permu¬ 
tation of those of the identity matrix. A permutation matrix is orthogonal. 

A square matrix A is symmetric if A T = A. A symmetric matrix has an 
eigenvalue-eigenvector decomposition of the form 


A = QEQ T 


where Q is orthogonal and E is (real) diagonal. The diagonal elements of E 
are the eigenvalues of A and the column vectors of Q are the eigenvectors of 
A. The Jth column vector of Q, say, q,, is associated with the jth eigenvalue, 
e, j, and satisfies the equation Aq, — e,/q,. The matrix A — e n I is singular. 
The eigenvalues of a symmetric n x n matrix A are unique. If a number l 
occurs m times among the n eigenvalues of A, then the m*dimensional sub¬ 
space spanned by the m eigenvectors (columns of Q ) associated with l is 
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uniquely determined and is called the eigenspace of A associated with the 
eigenvalue X. 

A symmetric matrix is positive definite if all its eigenvalues are positive. 
A positive definite matrix P is also characterized by the fact that x*Px > 0 
for all x 9 * 0 . 

A symmetric matrix S is nonnegative definite if its eigenvalues are non* 
negative. Such a matrix satisfies jfSx ;> 0 for all x & 0. 

For any m x n matrix A the matrix S = A T A is symmetric and nonnega¬ 
tive definite. It is positive definite if A is of rank n. 

An invariant space of a square matrix A is a subspace T such that x e T 
implies Ax e T. If 5 is a symmetric matrix, then every invariant space of S 
is the span of some set of eigenvectors of S, and conversely the span of any 
set of eigenvectors of 5 is an invariant space of S. 

A symmetric matrix P is a projection matrix if all its eigenvalues are either 
unity or zero. A matrix P is idempotent if P* — P [equivalently P(l — P) = 
0]. A matrix P is a projection matrix if and only if P is symmetric and 
idempotent. 

Let P„ xn be a projection matrix with k unit eigenvalues. Let T denote the 
fc-dimensional eigenspace of P associated with the unit eigenvalues. The 
subspace T is the (unique) subspace associated with the projection matrix P 
and P is the (unique) projection matrix associated with the subspace T. The 
subspace T is both the row space and column space of its associated projec¬ 
tion matrix P and is characterized by 


T — {x:Px — x} 


If Pis a projection matrix with associated subspace Tand ||Px|| = ||x||, 
then Px — x and x e T. Furthermore ||Px|| < ||x|| for x $ T and thus 
||P||=*1 unless P = 0. 

For any matrix A mXm the matrix A*A is the n x n projection matrix as¬ 
sociated with the row space of A, (/ — A*A) is the projection matrix as¬ 
sociated with the null space of A, and AA* is the m x m projection matrix 
associated with the range (column space) of A. 

Given any n-vector x the unique representation of x as x = t + u where 
t e Tand ae^is given by t — Px and u — {I — P)x. The vector t is the 
nearest vector to x in Tin the sense that || x — t |j = min{||x — t>|| \v e T}. 
The vector t is called the projection of x onto T. 

For a discussion of projection operators including proofs of most of the 
assertions above see pages 43-44 of Halmos (19S7). 

An arbitrary matrix A mXx has a singular value decomposition A = 
U mxm S mxm VZ xn where U and V are orthogonal and S is diagonal with non¬ 
negative diagonal elements. The diagonal elements of S (s,„ i — 1. k, 

where k — min [m, «]) are the singular values of A. This set of numbers is 
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uniquely determined by A. The number of nonzero singular values is the 
rank of A. Furthermore 

Mil = max,{jr M } and \\A\\, - (jfj sH)‘ /l 

which provides a simple proof of the two rightmost inequalities (A.3). 

Let A — USV T be a singular value decomposition of A mxm . Then eigen¬ 
value decompositions of A T A and AA T are given by A T A => V(S r S)V T and 
AA r = U(SS r )U T . 

The singular value decomposition provides information useful in the 
practical analysis of linear algebraic problems and is discussed further in 
Chapters 4,5,18,25, and 26. 


APPENDIX 



PROOF OF GLOBAL QUADRATIC 
CONVERGENCE OF THE QR ALGORITHM 


The purpose of this appendix is to give a proof of Theorem (18.5). The 
proof given is a more detailed version of that given in Wilkinson (1968a and 
1968b). Conclusion (a) of Theorem (18.5) is left as Exercise (18.46). 

The content of Lemma (B.l) will be found, with proof, in Wilkinson 
(1965a). 

(B.1) Lemma 

Let Abe an n x n symmetric tridiagonal matrix with diagonal terms 

a„ ..., a. and super - and subdiagonal terms b 2 .b„ where each 

b, is nonzero. Then the eigenvalues of A are distinct. 

We also need the following simple lemma whose proof is left as Exercise 
(B.52). Use will be made of notation introduced in the text and equations of 
Chapter 18 from the beginning through Eq. (18.4). 

(B.2) Lemma 

The elements of the matrices A k and the shift parameters a k are bounded 
in magnitude by | | A11, and the elements of the matrices (A k — o k I a ) and 
R k are bounded in magnitude by 21| A||,/or all k = 1,2,.... 

We must examine the basic operations of the QR method with shifts to 
establish properties of certain intermediate quantities and ultimately to esti¬ 
mate the magnitude of certain of the off-diagonal terms of the matrices A k . 
Denote the diagonal terms of the shifted matrix (A k — o k I.) by 

<*(*> = al*> - o k i = 1 ,.... n 

By the choice rule for a k [see Eq. (18.4)] it follows that the eigenvalue of the 
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lower right 2x2 submatrix of (A k — cr k Q which is closest to a<*> is zero. 
Thus 


(B.3) = (W 

and 


(B.4) 


m <: <£ ie\ i 


The orthogonal matrix Q k is determined in the form 
(B.5) 

where each/,*?,., is a rotation in the plane determined by the (i — l)$t and ith 
coordinate axes. The scalars c/* 1 and sj k) of the rotation matrices, as given in 
Lemma (3.4), define the rotation by means of the identity 


(B.6) 



0 



0 P\ k) 
0 0 


0 ‘ 
0 

4-i. 


i = 2 ,..., n 


Here each P! ki is defined as in Eq. (3.5) with c and s appropriately super- and 
subscripted. 

Following premultiplication of (A k — o k I m ) by the first i — 2 of the rota¬ 
tion matrices we shall have 


(B.7) • • • A k \(A k - c k I n ) 




rV 




0 

PV 

4 

qV 

r?» 





• 

• 

pI*-\ 

qft 

■ f}-2 




0 

x}i\ 

yi-\ 





6! k > 

W' bi k \ 


b' k \ a<i\ 6<« 

0 0 b lk) a< k > 


Premultiplying both sides of Eq. (B.7) by J l t k \. t shows, by means of Eq. (3.7) 
to (3.9), that the following recurrence equations hold. 
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(B.8) 

p£\ - 

= w,) 1 + wm* 

1 = 2,.. 

., n 

(B.9) 

P? = 

= x<« 



(B.10) 

C,<*> = 

Wi 

i = 2,.. 

. ,n 

(B. 11) 

S™ = 

_ W*’ 
pffi 

/ = 2," 


(B.12) 

0 = 

= + c} k >b}» 

i = 2,.. 

., n 

(B.13) 

*,<*> = 

= -s}*'y}i\ + cW 

1 = 2,.. 

. ,n 

(B.14) 

yP - 

= c}»b}*\ 

i = 2,.. 



The similarity transformation is completed by forming 
(B.15) A k+ 1 — R k Ql 4 - <f*/i 

= R k (A k \Y • ■ ■ + o*i. 

In this process the new off-diagonal elements are computed as 


(B.16) 


= *,<*> />,<*> 


i 



* • • 


$ 


n 


We wish to analyze the convergence of bl k + u ask—* oo. 

In the Eq. (B.17) to (B.19) we suppress the superscript (k) but write the 
superscript (k -f 1). Beginning with Eq. (B.16) with i = n we have 


(B.17) 


b { , k * u - S'P, 

= sjc, using (B.9) 

= sJi-s m y m ., + cAJ using (B. 13) 

= j„[- s„c n . A + V,] using (B. 14) 


— s n c n . t b H -|- ^4^ jc„_i J using (B.12) 

= J,j^— s K c k .ib„ + s *-i c *-ib*-i + 



sls*-jC»-2b K -ib m 
On -1 


using (B.3), (B.13), and (B.14) 


Eliminatingbetween Eq. (B.ll) and (B.16) gives 


(B.18) 


£<*+u _ s>-\b, 
*' 1 s , 


Then using Eq. (B.17) and (B.18) we obtain 

(B.19) br ««SV> - -(^) 
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From Eq. (B.4) expressed as | I < 1 and Eq. (B.17) we have 

(B.20) |6‘* +,> |<ieu k —1,2,... 

From Eq. (B.19) we see that 

(B.21) r*i* +, >ev , l<I«»«!*. I k — 1 , 2 ,... 

so that | b ( H k) bl*\ | converges to a limit L > 0 as k —* 

(B.22) Lemma The limit L is zero. 

Proof: Assume L> 0. Passing to the limit, as k —► oo, in both 


members of Eq. (B.19) yields 


(B.23) 

*«*> 

«i-i 

(B.24) 

1 1 

(B.25) 


(B.26) 



From Eq. (B.10) and (B.24) and the fact that the sequence {pi-M is 
bounded [Lemma (B.2)], we obtain 

(B.27) x<*J, - 0 

Thus from Eq. (B.13) and (B.14), 

(b. 28) - o 

Since Eq. (B.26) implies c ( ,i\ —* 0, it follows from Eq. (B.28) that 

(B.29) - 0 

From Eq. (B.25), (B.26), and (B.29) we have —*• 0, but since the 

sequence is bounded, this implies —► 0, contradicting the as¬ 

sumption that L > 0. This completes the proof of Lemma (B.22). 

(B.30) Lemma 

The sequence {| b ( n k) j}, k = 1,2,..., contains arbitrarily small terms. 

Proof: Let x > 0. By Lemma (B.2) the sequences {| bf |} and 
(| b ( m k Jt |] are bounded. Thus by Lemma (B.22) there exists an integer k for 
which either 


(B.31) 


I bii\ I < T 
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or 

(B.32) | Ai* 1 1 < r 

If Eq. (B.31) holds, then with Eq. (B.20) we have 

I *« +,, |<T 

It follows that for any r > 0 there exists an integer fc depending upon r such 
that 

This completes the proof of Lemma (B.30). 

(B.33) Lemma 

Jf x > 0 is sufficiently small and if k is such that | | < t, then 

| bi k) | < r for all k > k. 

Proof: Define S — min {| A, — Xj \ : i ^ j}. Note that 6 > 0 since 
the eigenvalues A, of A are distinct. 

Define 

(B.34) f(t) = 

Let t 0 > 0 be a number small enough so that 

(B.3S) T 0 < 

(B.36) /(To) < 1 

and 

(B.37) ittp- > 0 for 0 < ? < T 0 

Choose t, 0 < t < t 0 , and let k be an integer such that 
(B.38) ie’l<T 

For notational convenience we introduce 
(B.39) e = b™ 

If b ( n k) — 0, the algorithm has converged. Otherwise without loss of generality 
we may assume e > 0. 
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Since a lk) di k J t — e 1 we can write 





0 ' 

• 

(B.40) 

— &klm — 

B 

• 

• 

0 




e 



0 • • • 0 f 

f* 


Let fi„. t be the roots of the (n — 1) x (n — 1) symmetric matrix B, 

and A't,..., X„ be the roots of the shifted matrix A k — a k I m . By Theorem 
(S.l), the ordered eigenvalues of A„ — a k I H and those of 


B 0 

0 4r 

differ at most by e. Thus with a possible reindexing of the 4 we have the 
inequalities 



(B.41) 


\K~Mi / = 1 . n — 1 

X* “ glfrl ^ f 


From the identity 


Mi — Mi — 


X, 4- A? — X„ + X, — -f -gpr 

u a-1 u a-1 


we have 


(B.42) |4-ah _|/<,-A!|- 


l' 


e* 
Wi 
i= 1, 



Now using Eq. (B.41) on the second and third terms of the right side of 
inequality (B.42), observing that | X ( — X m | = | A/ — A, | > d, followed by use 
of Eq. (B.4) with b {k) = e on the fourth term of the right side of inequality 
(B.42) we have 

(B.43) —3f i — 1, ..., /i — 1 

At the next-to-last step in the forward triangularization of A k — a k I, the 
configuration in the lower right 2x2 submatrix is seen by Eq. (B.3), (B.7), 
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(B.14), and (B.39) to be 


(B.44) 


r*«. «£>, 


Wt 


Note that x£\ of Eq. (B.44) is a diagonal element of the (it — 1) x 
(it — 1) upper triangular matrix resulting from premultiplication of the 
matrix B of Eq. (B.40) by (it — 2) rotation matrices. Thus by Eq. (6.3) 
and (B.43) 


(B.45) 


1*2?! I ^ min 10,1 - 3e 


Completing the similarity transformation we have, from Eq. (B.3), (B.9), 
(B.13), and (B.17) with y£>, - = c«2,e. 


(B.46) 




= («» - tfy&W 


Now from Eq. (B.l 1) and the inequality (B.43) 


(B.47) 


,?>l = ^ T=r% 


Finally, using Eq. (B.46) and the inequality (B.47) we have 


(B.48) 




The inequality in Eq. (B.48) shows that if | a™, | is bounded away from 
zero, then the convergence ultimately is cubic. However, all we know general¬ 
ly from Eq. (B.4) is that | dll’, | ^ e, and therefore 


(B.49) 


I bj*+»\£ el 


or, recalling Eq. (B.34) and (B.39), 

(B.50) [ *?♦» | ^ [OT 1 + - M*W) 

From conditions (B.36) to (B.38) we have/^’) < 1 and thus 


(B.51) 
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By induction it follows that inequalities (B.50) and (B.51) with k replaced by 
k + l hold for all / = 0,1,_This completes the proof of Lemma (B.33). 


Lemma (B.33) implies that bi 1 —* 0 as / —* oo, which is conclusion (b) 
of Theorem (18.5). Since bj? —»0 and the inequality in Eq. (B.50) holds for 
all sufficiently large k the quadratic convergence asserted in the final con¬ 
clusion (c) of Theorem (18.5) is also established. 

We conclude this appendix with these remarks: 


1. The shifts <r* do not need to be explicitly subtracted from each of the 
diagonal terms of A k when forming A t+i . Discussion of this as it applies to 
computation of the singular value decomposition is given in Chapter 18. 

2. In practice the iteration procedure for each eigenvalue is terminated 
when the terms bp are “zero to working accuracy.” This can be defined in a 
variety of ways. One criterion for this is given in Chapter 18 as it concerns the 
numerical aspects of the singular value decomposition. 

3. The proof we have given assumes that n 3. For n = 2 it is easy to 
see that if 


A = A, 


(*P 


bP\ 

aP) 


then performing one shifted QR transformation as given in Eq. (18.1) to 
(18.4) gives 

/«?> bP\ 

2 Up «?’/ 


with b{ 2) = 0. Thus A % is diagonal and the eigenvalues of A have been com¬ 
puted. 

4. More generally if an actual eigenvalue A of ^ instead of Or of Eq (18.4) 
were used as a shift, then the matrix A k + , would break up after the next QR 
sweep. In fact, bj?* u — 0 and a'* +,) = X. To verify this note that since A„ — 
XI, , is rank deficient at least one of the diagonal terms pP of the triangular 
matrix R k must be zero. From Eq. (B.8), pP > 0 for / = 1,..., n — 1 and 
therefore it must be pP that is zero. Upon completion of the similarity trans¬ 
formation and translation of Eq. (B.15) while using Eq. (B.17) we see that 
= sPpP — 0, and therefore — X. 


EXERCI8E 

(B.52) Prove Lemma (B.2). 


APPENDIX description and use of 

C FORTRAN CODES FOR 

SOLVING PROBLEM LS 


Introduction 

Throughout this book, attention has focused primarily on the mathematical 
and algorithmic descriptions of various methods for obtaining solutions 
to Problem LS (Chapter 1) or its variants such as Problem LSE (20.1) 
and Problem LSI (23.1). These algorithms can be implemented in any of 
several well-known computer source languages such as Fortran, C, or Ada. 
Accomplishing this can be expensive and time consuming. For this reason 
we have prepared Fortran source codes for some of the algorithms. All but 
one of the codes are written to conform to the Fortran 77 standard. The 
exception is BVLS, which conforms to the Fortran 90 standard. 

These codes can be obtained from the NETLIB site on the Internet. 
NETLIB has become the preeminent mechanism in the known universe 
for collecting and distributing public-domain mathematical software. For 
background information on NETLIB see Dongarra and Grosse (1987), Don- 
garra, Rowan, and Wade (1995), and Grosse (1995) in Appendix D. The 
current alternate Internet addresses for NETLIB for general ftp and e-mail 
access are netlib.org and netlibQresearch.att.com. A mirrored ad¬ 
dress in Norway is netlib.no. The URL for World Wide Web access to 
NETLIB is http://www.netlib.org. 

The 1974 versions of these programs (except PROG7 and BVLS) were 
compiled and executed successfully on the UNIVAC 1108 and IBM 360/75 
computer systems at the Jet Propulsion Laboratory, on the IBM 360/67 at 
Washington State University, on the CDC 6500 at Purdue University, on 
the CDC 6600 and 7600 at the National Center for Atmospheric Research, 
and on the CDC 7600 at Aerospace Corporation. We wish to express our 
appreciation to Drs. T. J. Aird, Alan Cline, and H. J. Wertz and Mr. C. 
T. Verhey for their valuable cooperation in making these runs on CDC 
computers. 

Summary of subprograms implemented 

The subprograms implemented are 

• HFTI. Implements Algorithm HFTI of Chapter 14. Calls H12 and 
DIFF. 
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• SVA. Implements singular value analysis and Levenberg-Marquardt 
analysis as described in Chapters 18 and 25. Produces printed output 
of quantities of interest. Calls SVDRS and MFEOUT. 

• SVDRS. Computes the singular value decomposition as described in 
Chapter 18. Called by SVA. Calls H12 and QRBD. 

• QRBD. Computes the singular value decomposition of a bidiagonal 
matrix as described in Chapter 18. Called by SVDRS. Calls DIFF 
and Gl. 

• BNDACC and BNDSOL. Implements the band-limited sequential 
least squares algorithm of Chapter 27, Section 2. Calls H12. 

• LDP. Solves the least distance programming problem as described 
in Chapter 23. Calls NNLS and DIFF. 

• NNLS. Computes a least squares solution, subject to all variables 
being nonnegative, as described in Chapter 23. Called by LDP.Calls 
H12, DIFF, and Gl. 

• H12. Constructs and applies a Householder transformation as de¬ 
scribed in Chapter 10. Called by HFTI, SVDRS, NNLS, and 
BNDACC. 

• Gl and G2. Constructs and applies a Givens rotation as described 
in Chapter 10. Gl is called by QRBD and NNLS. The code repre¬ 
senting Algorithm G2 appears inline in QRBD and NNLS. 

• MFEOUT. Prints a two-dimensional array in a choice of two for¬ 
mats. Called by SVA. 

• GEN. Generates a sequence of numbers for use in constructing test 
data. Used by PROG1, PROG2, and PROGS. 

• DIFF. Computes the difference between two floating point argu¬ 
ments. Called by HFTI, QRBD, LDP, and NNLS to support a 
method of comparing computed quantities against the precision limit 
of the host computer. 

• BVLS. A Fortran 90 subroutine that computes a least squares solu¬ 
tion, subject to all variables having upper and lower bounds. 
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Summary of main programs implemented 

Seven main programs are implemented to demonstrate the various algo¬ 
rithms and the usage of the subprograms. 

• PROGl. Demonstrates Algorithms HFT, HSl, and COV of Chap¬ 
ters 11 and 12. Calls H12 and GEN. 

• PROG2. Demonstrates Algorithms HFTI and COV of Chapters 
14 and 12, respectively. Calls HFTI and GEN. 

• PROG3. Demonstrates the singular value decomposition algorithm 
of Chapter 18. Calls SVDRS and GEN. 

• PROG4. Demonstrates singular value analysis including computa¬ 
tion of Levenberg-Marquardt solution norms and residual norms as 
described in Chapters 18, 25, and 26. Calls SVA and reads test data 
from the Hie DATA4. The Hie DATA4 contains the data listed in 
Table 26.1, and the computation produces the results shown in Figure 
26.1. 

• PROG5. Demonstrates the band-limited sequential accumulation 
algorithm of Chapter 27, Sections 2 and 4. The algorithm is used 
to Ht a cubic spline (with uniformly spaced breakpoints) to a table 
of data. This example is discussed in Chapter 27, Sections 2 and 4. 
Calls BNDACC and BNDSOL. 

• PROG6. Computes the constrained line-fitting problem given as an 
example in Section 7 of Chapter 23. The program illustrates a typical 
usage of the subroutine LDP, which in turn uses NNLS. PROG6 
also calls SVDRS. 

• PROG7. A Fortran 90 program that demonstrates the use of BVLS. 

Discussion of PROGl, PROG2, and PROG3 

The program PROGl is an illustration of the simplest method of solv¬ 
ing a least squares problem described in this book. It implements the 
Algorithms HFT, HSl, and COV of Chapters 11 and 12. This code may 
give unacceptable answers if the matrix is nearly rank-deficient, so beware: 
no checking for nearly rank-deficient matrices is made in PROGl. 

Methods coping with problems having these nearly rank-deficient ma¬ 
trices are illustrated by PROG2 and PROG3, which demonstrate, re¬ 
spectively, Algorithms HFTI (Chapter 14) and SVD (Chapter 18). The 
reader who is solving least squares problems with near rank deficiencies 
(highly correlated variables) would be advised to use code based on one 
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of these more general programs. The program PROG2 also implements 
Algorithm COV (Chapter 12} in the cases that are of full pseudorank. 

The programs PROG2 and PROGS each use an absolute tolerance 
parameter r for pseudorank determination. In PROG2, r is compared 
with diagonal elements of the triangular matrix resulting from Householder 
triangularization, while in PROGS, r is compared with the singular values. 

Each of the three main programs, PROG1, PROG2, and PROG3, 
uses the FUNCTION subprogram GEN to generate the same set of 
36 test cases. For the first 18 cases the data consist of a sequence of 
integers between —500 and 500, with a period of 10. This short period 
causes certain of the matrices to be mathematically rank-deficient. All the 
generated matrices have norms of approximately 500. 

For the first 18 cases PROG2 sets the absolute tolerance r = 0.5. 
In PROGS a relative tolerance p is set to 10~ 3 . Then for each case the 
absolute tolerance is computed as r — ps\ where 8\ is the largest singular 
value of the test matrix A. Recall that s\ = j|A)|. 

The second set of 18 cases are the same as the first 18 except that 
“noise,” simulating data uncertainty, is added to all data values. The 
relative magnitude of this “noise” is approximately u = 10“ 4 . For these 
cases PROG2 sets r = 10 va where a is preset to 500 to represent the 
approximate norm of the test matrices. The program PROGS sets p = 10 v 
and then, for each test matrix A, computes r = p\\A\\. 

This method of setting r, particularly in the second 18 cases, is intended 
to stress the idea of choosing r on the basis of a priori knowledge about 
the size of data uncertainty. 

If the reader executes these programs, he or she will find that in the 
pseudorank-deficient cases the solutions computed by PROG2 and 
PROGS are similar and of order of magnitude unity, while those com¬ 
puted by PROG1 contain very large numbers that would probably be 
unacceptable in practical applications. For example, for the test case 
with m = 7,n = 6, and a noise level of zero, the values computed by 
PROG1, PROG2, and PROG3 for the first component of the solution 
vector on a computer with precision of about 10~ 8 were 0.237e8, 0.1458, 
and 0.1458, respectively. With a relative noise level of 10~ 4 , the values 
computed were —0.4544e4, 0.1470, and 0.1456. In these cases PROG1 
reported a pseudorank of 6, whereas PROG2 and PROG3 reported a 
pseudorank of 4. 

Machine and Problem Dependent Tolerances 

In order to avoid storing the machine precision constant r\ in various sub¬ 
programs, tests of the general form “Ifflhj > r/|x|)” have been replaced by 
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the test “If ((a: + ft) —x^O).” To obtain the intended result, it is essential 
that the sum x + h be truncated (or rounded) to imprecision before com¬ 
puting the difference (x + h) — x. If the expression (x + h) — x were written 
directly in Fortran code, there is the possibility that the intended result 
would not be obtained due to an optimizing compiler or the use of an ex¬ 
tended length accumulator. To circumvent these hazards a FUNCTION 
subprogram, DIFF, is used so that the test is coded as “If(DIFF(x+h, x) 
^ 0).” In newer languages, such as Fortran 90, ANSI C, and Ada, access to 
a machine precision parameter is provided by the language, so techniques 
such as this are not needed. 

The call statement for HFTI includes a parameter, TAU, which is 
used as an absolute tolerance in determining the pseudorank of the matrix. 
It is intended that the user set the value of this parameter to express 
information about the accuracy of the data rather than to express the 
machine’s arithmetic precision. 

Conversion of Fortran 77 Codes Between REAL and 
DOUBLE PRECISION 

All of the variables in these codes are declared. All constants are intro¬ 
duced in PARAMETER statements. All intrinsic functions are used by 
their generic names. As a consequence it should be possible to change 
these Fortran 77 codes between REAL and DOUBLE PRECISION just by 
making changes in the declaration part of the codes. 

The Algorithm BVLS 

The NNLS algorithm first appeared in the 1974 edition of this book. A 
number of readers of the book have informed us that they have used NNLS 
successfully in a variety of applications. NNLS has also been incorporated 
in a number of commercial software packages. 

Since 1974 we developed an algorithm and code for the Bounded Vari¬ 
ables Least Squares problem, BVLS. This is a generalization of the NNLS 
algorithm in which the constraints on the variables are a* < x* < fo rather 
than the constraints x* > 0 of NNLS. The structure of the BVLS algorithm 
is essentially the same as NNLS, with additional details to deal with the 
two-sided inequalities. 


User’s Guides for the Subprograms 
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HFTI 

USER'S GUIDE TO HFTI: SOLUTION OP THE LEA8T SQUARES 
PROBLEM BY HOUSEHOLDER TRANSFORMATIONS 

Subroutines Called HI 2, DIFF 
Purpose 

This subroutine solves a linear least squares problem or a set of linear 
least squares problems having the same matrix but different right-side vectors. 
The problem data consists of an M x N matrix A, an M x NB matrix B, 
and an absolute tolerance parameter t. The N B column vectors of B represent 
right-side vectors b, for NB distinct linear least squares problems. 

Ax, jg b, y = 1.....NB 

This set of problems can also be written as the matrix least squares problem: 

AX~B 

where X is the N x NB matrix having column vectors x,. 

Note that if B is the M x M identity matrix, then X will be the pseudo- 
inverse of A. 

Method 

This subroutine first transforms the augmented matrix [A : B] to a matrix 
(A: C] using premultiplying Householder transformations with column inter¬ 
changes. All subdiagonal elements in the matrix R are zero and its diagonal 
elements satisfy |r„| ^ |r l+M+ , |, / = 1,..., / — 1, where / = min {M, N}. 

The subroutine will set the pseudorank KRANK equal to the number of 
diagonal elements of R exceeding t in magnitude. Minimal length solution 
vectors X„ j = , NB, will be computed for the problems defined by 

the first KRANK rows of [A: C]. 

If the relative uncertainty in the data matrix B is p, it is suggested that r 
be set approximately equal to p || A ||. 

For further algorithmic details, see Algorithm HFTI in Chapter 14. 

Usage 

DIMENSION A(MDA,»,), {B(MDB, n t ) or B(m,)}, RNORM(n 2 ). 

H(*»,). G(i.,) 

INTEGER IP(»,) 

CALL HFTI (A, MDA, M, N, B, MDB, NB.TAU, KRANK, RNORM, 
H, Q, IP) 

The dimensioning parameters must satisfy MDA M, n, ;> N, MDB 
^ max {M, N}, m, ^ max (M, N}, and n t ^ NB. 

The subroutine parameters are defined as follows: 
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A{,), MOA, M, N The array A(. ) initially contains the M x N matrix 

A of the least squares problem AX S B. The first 
dimensioning parameter of the array A( , ) is MDA, 
which must satisfy MDA > M. Either M ;> N or 
M < N is permitted. There is no restriction on the 
rank of A. The contents of the array A(, ) will be 
modified by the subroutine. See Fig. 14.1 for an exam* 
pie illustrating the final contents of A(,). 

B( ), MDB, NB If NB = 0 the subroutine will make no references 

to the array B( ). If NB > 0 the array B( ) must 
initially contain the M x NB matrix B of the least 
squares problem AX = B and on return the array 
B( ) will contain the N x NB solution matrix Jt. 
If NB > 2 the array B( ) must be double sub¬ 
scripted with first dimensioning parameter MDB > 
max {M, N}. If NB = 1 the array B( ) may be 
either doubly or singly subscripted. In the latter case 
the value of MDB is arbitrary but some Fortran 
compilers require that MDB be assigned a valid 
integer value, say MDB =1. 

TAU Absolute tolerance parameter provided by user for 

pseudorank determination. 

KRANK Set by the subroutine to indicate the pseudorank 

of A. 

RNORM( ) On exit, RNORM(J) will contain the euclidean 

norm of the residual vector for the problem defined 
by the y'th column vector of the array B( ,) for j — 

1.NB. 

Arrays of working space. See Fig. 14.1 for an example 
illustrating the final contents of these arrays. 

Array in which the subroutine records indices describ¬ 
ing the permutation of column vectors. See Fig. 14.1 
for an example illustrating the final contents of this 
array. 

Example of Usage 

See PROG2 for an example of the usage of this subroutine. 


H( ). G( ) 
IP( ) 
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SVA 

USER'S GUIDE TO SVA: SINGULAR VALUE ANALYSIS 

Subroutines Called SVDRS, MFEOUT 
Purpose 

The subroutine SVA uses subroutine SVDRS to obtain the singular 
value decomposition of a column scaled matrix X = AD and the associated 
transformation of the right-side vector b of a least squares problem Ax ~ b. 
The subroutine SVA prints quantities derived from this decomposition to 
provide the user with information useful to the understanding of the given 
least squares problem. 

Method 

The user provides an m x n matrix A and an m-vector b, defining a least 
squares problem Ax — b. The user selects one of three options regarding 
column scaling described below in the definition of the parameter ISCALE. 
This selection defines an n x n diagonal matrix D. 

Introducing the change of variables 


* = Dy 


the subroutine performs a singular value analysis of the least squares problem 


where 


Ay = b 
X — AD 


The subroutine SVA uses the subroutine SVDRS to compute the singu 
lar value decomposition 

X = usv T 

and the transformed right-side vector 


U T b 


Denote the (ordered) singular values of X, i.e., the diagonal elements of S, 
by j,, ..., s B . Let h denote the index of the last nonzero singular value and 
define pt = min (m, n). 

Compute the vector 


with components 


P = S+g 




i — i,.,., a 

i = n + 1. n 
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The Jfcth candidate solution vector y 1 *’ is defined as 


/«> = o 



where Vj denotes the Jth column vector of V. Reverting to the original vari¬ 
ables we have the corresponding candidate solutions 

x m = Dy t» k = 0,... t ft 

The quantities 

pl — II gf k = 0 . ft 

l-T+l 

are computed. For k ^ n the quantity pl is the sum of squares of residuals 
associated with the Jfcth candidate solution; i.e., /»* = ||4 — ^jc‘* > || j . 

It is possible that the m x (« + 1) data matrix [A : b] provided to this 
subroutine may be a compressed representation of a least squares problem 
involving more than m rows. For example, [A: b] may be the triangular 
matrix resulting from sequential Householder triangularization of a large set 
of data as described in Chapter 27. The user provides an integer MDATA 
that specifies the number of rows of data in the original problem. Of course, 
if [A: 6] is the original data, then MDATA and M should be set to the same 
value. The number MDATA is used in computing 



Under appropriate statistical hypotheses on the data [A : 4], the number 
o k can be interpreted as an unbiased estimate of the standard deviation of 
errors in the data vector 4. 

Adapting Eq. (25.47) and (25.48) to the notation of this Subroutine User’s 
Guide, we compute the following quantities, which may be used for a Leven- 
berg-Marquardt analysis: 

.;=iiKi)ip=gAt7^p)‘ 
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This subroutine has the capability of printing output information that 
is organized into six blocks. The user has the option, via the argument 
KPVEC, to select individually which of these blocks are to be printed. 
The contents of the blocks are as follows: 

1. The quantities M, N, MDATA, identification of the scaling option 
used, and the diagonal elements of the scaling matrix D . 

2. The matrix V, multiplied by 10 4 to facilitate scanning for large and 
small elements. 

3. Singular values and related quantities: s* for k = 1,..., p; pk , s*" 1 , 
9k>9k 2 , for k = 1 ,.. .,n; and p* 2 and for k = 0,. . .,n. 

4. Solution norm versus residual norm at different singular value cutoffs: 

lb (fc) ll>P*.logio ll» ( * ) ll. and log 10 p fc , for k = 1 ,... ,n. 

5. Levenberg-Marquardt analysis: A,i/A,tfA,log l0 A,log 10 i/A, and log 10 
u?\ for a sequence of 21 values of A ranging from 10ai to Sn/lO in 
increments that are uniformly spaced in log 10 A. 

6. Candidate solutions at different singular value cutoffs: for k = 

1,«.., w« 


Usage 

INTEGER MDA, M, N, MDATA, KPVEC(4), ISC ALE 
REAL A(MDA,m),B(m x ),SING(n x ),D(n x ) t WORK(n 2 ) 
CHARACTER NAMES(n x ) * (lennam) 

CALL SVA(A, MDA, M, N, MDATA, B, SING, KPVEC, NAMES, 
ISC ALE, D, WORK) 

The dimension parameters must satisfy MDA > max(M, N), 
ni > N, mi > M, n 2 > 2xN, lennam > 1. 

The subroutine parameters are defined as follows: 


A(,), MDA, M, N 
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The array A(,) initially contains the M x N matrix A of the 
least squares problem Ax S* b. Either M>NorM<Nis 
permitted. The first dimensioning parameter of the array A(,) 
is MDA, which must satisfy MDA > max(M, N). On output 
the fcth candidate solution will be stored in the first N 
locations of the fcth column of A(,) for k « 1,..., min(M, N). 

MDATA 

The number of rows in the original least squares problem, which 
may have had more rows than [A : 6]. MDATA is used only 
in computing the numbers k = 0,..., n. 


B( ) 


The array B( ) initially contains the M-vector 6 of the least 
squares problem Ax mb. On output the M-vector g is stored 
in the array B( ). 

SING( ) 

On return the singular values of the scaled matrix A are stored 
in descending order in SING(i), i ** 1,..., min(M, N). If M 
< N, SING(M+1) through SING(N) will be set to zero. 

KPVEC( ) 

Option array controlling report generation. If KPVEC(l) = 0, 
default settings will be used, producing the full report, sending 
it to the standard system output unit, formatted with a max¬ 
imum line length of 79. If KPVEC(l) = 1, the contents of 
KPVEC(i), i = 2,..., 4, set options for the report as follows: 

KPVEC(2) 

The decimal representation of KPVEC(2) must be at most 6 
digits, each being 0 or 1. The decimal digits will be interpreted 
as independent on/off flags for the 6 possible blocks of the report 
that were described above. Examples: 111111, which is the 
default value, selects all blocks. 101010 selects the 1st, 3rd, 
and 5th blocks. 0 suppresses the whole report. 

KPVEC(3) 

Define UNIT = KPVEC(3). If UNIT = -1, which is the 
default value, output will be written to the output unit, 
i.e., the standard system output unit. If UNIT > 0, UNIT will 
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be used as the output unit number. The calling program is 
responsible for opening and/or closing the selected output unit 
if the host system requires these actions. 

KPVEC(4) 

Determines the width of blocks 2, 3, and 6 of the output re¬ 
port. Define WIDTH = KPVEC(4). The default value is 79. 
Each output line will have a leading blank for Fortran “carriage 
control” with line widths as follows: Output blocks 1, 4, and 
5 always have 63, 66, and 66 character positions, respectively. 
Output blocks 2 and 6 will generally have at most WIDTH char¬ 
acter positions. One output line will contain a row number, a 
name from NAMES( ), and from one to eight floating-point 
numbers. The space allocated for a name will be that needed 
for the longest name in NAMES( ), which may be less than 
the declared length of the elements of NAMES( ). The line 
length will only exceed WIDTH if this is necessary to accom¬ 
modate a row number, a name, and one floating-point number. 
Output block 3 will have 69 character positions if WIDTH < 
95 and otherwise will have 95 character positions. 

NAMES( ) 

NAMES (j), for j = 1,..., N, may contain a name for the jth 
component of the solution vector. If NAMES(l) contains only 
blank characters, it will be assumed that no names have been 
provided, and this subroutine will not access the NAMES ( ) 
array beyond the first element. 

ISCALE 

Set by user to 1,2, or 3 to select the column scaling option. 

1. The subroutine will use identity scaling and ignore the D( ) 
array. 

2. The subroutine will scale nonzero columns of A to have 
unit Euclidean length, and will store reciprocal lengths of the 
original nonzero columns in D()» If column j of A has only 
zero elements, D (j) will be set to one. 

3. User supplies column scaling factors in D( ). The subroutine 
will multiply column j of A by D (j) and remove the scaling 
from the solution at the end. 


D() 
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Usage of D( ) depends on ISCALE as described above. When 
used, its length must be at least N. 

WORK( ) 

Scratch work space of length at least 2xN. 


Example of Usage 


See PROG4 for an example of the usage of this subroutine. 
Output printed by this subroutine is illustrated in Fig 26.1. 


SVDRS 

USER’S GUIDE TO SVDRS: SINGULAR VALUE 
DECOMPOSITION OF PROBLEM LS 


Subroutines Called H12, QRBD 
Purpose 

Given an M x N matrix A and an M x NB matrix B, this subroutine 
computes the singular values of A and also computes auxiliary quantities 
useful in analyzing and solving the matrix least squares problem AX & 
B. Denoting the singular value decomposition of A by A = USV T , this 
subroutine computes 5, V, and G = L 

To complete the solution of the least squares problem AX =* B, the 
user must first decide which small singular values are to be treated as zero. 
Let S + denote the matrix obtained by transposing S and reciprocating 
the significant singular values and setting the others to zero. Then the 
solution matrix X can be obtained by computing P = S+G and X = VP. 
Either M>NorM<Nis permitted. Note that if B = /, then X is the 
pseudoinverse of A. 

Input 

The matrices A and B and their dimensioning parameters are input. 
Output 

The matrices V, G = C^B, and the diagonal elements of S are output in 
the arrays A(,), B(,), and S(), respectively. 

Usage 
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INTEGER MDA, M, N, MDB, NB 

REAL A(MDA,ni ),B(MDB,n 2 ) or 8 ( 03 ), WORK(n 4 ) 

CALL SVDRS (A, MDA, M, N, B, MDB, NB, S, WORK) 

The dimensioning parameters must satisfy MDA > max(M, N), 
rii > N, MDB > M, r »2 > NB, mi > M, 113 > N, and 
«4 > 2 x N. 

The subroutine parameters are defined as follows: 

A(,),MDA,M,N 

The array A(,) is a doubly subscripted array with first dimen¬ 
sioning parameter equal to MDA. The array A(,) initially 
contains the M x N matrix A with A(i, j) := a On output 
the array A( 9 ) contains the NxN matrix V with A(i, j) := 

Vij. Either M>NorM<Nis permitted. 

B( ),MDB,NB 

The value NB denotes the number of column vectors in the ma¬ 
trix B. If NB = 0, B( ) will not be referenced by this subrou¬ 
tine. If NB > 2, the array B(,) should be doubly subscripted 
with first dimensioning parameter equal to MDB. If NB = 1 , 
then B( ) will be used as a singly subscripted array of length 
M. In this latter case the value of MDB is arbitrary but for 
successful functioning of some Fortran compilers it must be set 
to some acceptable integer value, say, MDB = 1. The contents 
of the array B(,) are initially 

B(a, j) := bijy i = 1,—,M, j s* 1,...,NB 

or B(i) := i = 1,...,M. 

At the conclusion B(*, j) := g#, i = 1,..., M, j = 1,..., NB 
or B (i) := gi, i = 1,...,M. 



On conclusion the first N locations contain the ordered singular 
values of the matrix S(l) > S(2) > • • • > S(N) > 0. 

WORK( ) 


This is working space of size 2 x N. 
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Error Message 

In subroutine QRBD the off-diagonal of the bidiagonal matrix with 
smallest nonzero magnitude is set to the value zero whenever the iteration 
counter reaches 10 * N. The iteration counter is then reset. The results 
are therefore still likely to result in an accurate SVD. This is reported 
to SVDRS with the output parameter IPASS = 2. Subroutine SVDRS 
then prints the message 

FULL ACCURACY NOT ATTAINED IN BIDIAGONAL SVD 

Example of Usage 

See PROG3 and the subroutine SVA for examples of usage of this 
subroutine. 


QRBD 


USER'S GUIDE TO QRBD: SINGULAR VALUE 
DECOMPOSITION OF A BIDIAGONAL MATRIX 


Subroutines Called G1, G2, and DIFF 
Purpose 

Compute the singular value decomposition of an N x N bidiagonal 
matrix 

B = USV 7 


This subroutine implements the special QR iteration described in Chapter 18. 
The output of the subroutine consists of the singular values of B (the diagonal 
elements of the matrix 5) and the two matrix products Vf and U T C where 
V and C are given matrices of dimensions NRV x N and N x NC, respec¬ 
tively. 
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Usage 

DIMENSION D(n t ), E(n,), V(MDV, «,). C(MDC. m x ) 

CALL QRBD (IPASS. D, E. N. V. MDV, NRV, C, MDC, NC) 

The dimension parameters must satisfy w, ^ N, m, ^ NC, MDV ;> 

NRV, MDC ^ N. 

The subroutine parameters are defined as follows: 

IPASS This integer flag is returned with either of the values 

1 or 2. 

IPASS * 1 signifies that convergence of the QR 

IPASS = 2 denotes a loss of accuracy in the sin¬ 
gular values. Convergence was not attained after 
10 * N QR sweeps. This resulted in setting the 
smallest nonzero offdiagonal to zero. From Theo¬ 
rem (5.7) this may perturb the singular values by a 
value larger than the usual value, tj || A ||. 

D< ) The array D( ) initially contains the diagonal ele¬ 

ments of the matrix B. 

D(l) := b tt , I * 1.N 

On return the array D( ) contains the N (nonnega¬ 
tive) singular values of B in nonincreasing order. 

E( ) The array E( ) initially contains the superdiagonal 

elements of the matrix B. 

E(1) := arbitrary. 

E(l) := I - 2,.... N. 

The contents of the array E( ) are modified by the 
subroutine. 

N Order of the matrix B. 

V(.), MDV, NRV If NRV ^ 0, the parameters V(,) and MDV will 

not be used. If NRV ^ 1, then MDV must satisfy 
MDV ^ NRV. The first dimensioning parameter of 
V(, ) must be equal to MDV. The array V(,) 
initially contains an NRV x N matrix V and on 
return this matrix will be replaced by the NRV x N 
product matrix VP. 

C(,), MDC, NC If NC <; 0, the parameters C(,) and MDC will not 

be used. 

If NC ;> 2, then MDC must satisfy MDC > N. 
The first dimensioning parameter of the array C(,) 
must be equal to MDC. 
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If NC = 1, the parameter MDC can be assigned 
any value, say, MDC = 1. In this case the array 
C(,) may be either singly or doubly subscripted. 
When NC 1, the array C(,) must initially con* 
tain an N x NC matrix C. On output this matrix 
will be replaced by the N x NC product matrix 0 * 0 . 


Example of Usage 

This subroutine is used by the subroutine SVDR8. 


BNDACC, BNDSOL 


USER‘8 GUIDE TO BNDACC AND BND80L: SEQUENTIAL 
PROCESSING OF BANDED LEAST SQUARES PROBLEMS 


Subroutine Called H12 
Purpose 

These subroutines implement the algorithms given in Section 2 of Chapter 
27 for solving Ax~b where the matrix A has bandwidth NB. The four 
principal parts of these algorithms are obtained by the following CALL 
statements: 


CALL BNDACC(...) 
CALL BNDSOL(1....) 

CALL BNDSOL(2,...) 

CALL BND80L(3,...) 


Introduce new rows of data. 

Compute solution vector and norm of residual 
vector. 

Given a vector d, solve iP 'y = d for y {see Eq. 

(27.22) ]. 

Given a vector w, solve Rc = w for c [see Eq. 

(27.23) ]. 


The dots indicate additional parameters that will be specified in the following 
paragraphs. 


Usage of BNDACC 

DIMEN8ION G(MDG,»,) 

The dimensioning parameters must satisfy MDG ^ p [see Eq. (27.9) for 
the definition of p t] and a, ^ NB +1. 

The user must set IP - 1 and IR ■ 1 before the first call to BNDACC. 
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The subroutine BNDACC is to be called once for each block of data 
[C,: b,] to be introduced into the problem. See Eq. (27.14) and Fig. 27.2. For 
each block of data the calling program must assign values to MT and JT and 
copy the [MT x (NB + 1)]>array of data [C,: 6 f ] into rows IR through 
IR + MT — 1 of the working array G( ,) and then execute the statement 

CALL BNDACC (G, MDG, NB, IP, IR, MT. JT) 

The subroutine parameters are defined as follows: 

G(, ) The working array. See Eq. (27.15) to (27.17) and adjacent text. 

MDG Set by user to indicate the number of rows in the array G(,). 

NB Set by user to indicate the bandwidth of the data matrix A. See 

text following Eq. (27.14), 

IP Must be set to the value 1 before the first call to BNDACC. Its 
value is subsequently controlled by BNDACC. 

IR Index of first row of G( ,) into which user is to place new data. 
The variable IR is initialized by the user to the value 1 and sub¬ 
sequently updated by BNDACC. It is not to be modified by the 
user. 

MT Set by user to indicate the number of new rows of data being 
introduced by the current call to BNDACC. See Eq. (27.14). 

JT Set by user to indicate the column of the submatrix A, that is 
identified with the first column of the array C,. This parameter 
JT has the same meaning as/, of Eq. (27.14). 

Usage 0 /BND 8 OL 

This subroutine performs three distinct functions selected by the first 
parameter MODE, which may have the values 1,2, or 3. 

The statement CALL BNDSOL (1,...) may be executed after one or 
more calls to BNDACC to compute a solution for the least squares problem 
whose data has been introduced and triangularized by the calls to BNDACC. 
The problem being solved is represented by Eq. (27.11). This statement also 
computes the number | e\ of Eq. (27.6), which has the interpretation given in 
Eq. (27.12). The number |e| will be stored in RNORM. 

The computation performed by CALL BND80L (1,. ..) does not alter 
the contents of the array G(,) or the value of IR. Thus, after executing 
CALLBNDSOL(1,...), more calls can be made to BNDACC to introduce 
additional data. 

The statement CALL BNDSOL ( 2 ,...) may be used to solve KFy — d 
and the statement CALL BND80L (3,...) may be used to solve Rc — w. 
These entries do not modify the contents of the array G(, ) or the variable 
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IR. The variable RNORM is set to zero by each of these latter two state¬ 
ments. 

The primary purpose of these latter two statements is to facilitate compu¬ 
tation of columns of the covariance matrix C — (R T R)~ 1 as described in 
Chapter 27. To compute c Jt the Jth column of C, one would set d — e Jt the 
yth column of the identity matrix. After solving R T y — d, set w — y and 
solve Rcj — w. This type of usage is illustrated in the Fortran program 
PROGS. 

The appropriate specification statement is 

DIMENSION G(MDG, it,). X(» a ) 

The dimensioning parameters must satisfy MDG ^ n [see Eq. (27.9) 
for the definition of ft], /», ^ NB +1, and n a ;> N. The CALL statement is 

CALL BNDSOL(MODE. G, MDG, NB, IP, IR, X. N, RNORM) 

The subroutine parameters are defined as follows: 

MODE Set by the user to the value 1, 2, or 3 to select 

the desired function as described above. 

G(,), MDG, NB, IP. IR These parameters must contain values as they 

were defined upon the return from a preceding 
call to BNDACC. 

X( ) On input, with MODE ■ 2 or 3, this array 

must contain the N-dimensional right-side vec¬ 
tor of the system to be solved. On output with 
MODE ■ 1, 2, or 3 this array will contain 
the N-dimensional solution vector of the appro¬ 
priate system that has been solved. 

N Set by user to specify the dimensionality of the 

desired solution vector. This causes the sub¬ 
routine BNDSOL to use only the leading N x 
N submatrix of the triangular matrix currently 
represented in columns 1 through NB of the 
array G( ,) and (if MODE “ 1) only the first 
N components of the right-side vector currently 
represented in column NB 4-1 of the array 
G(r). 

If any of the first N diagonal elements are zero, 
this subroutine executes the Fortran statement 
STOP after printing an appropriate message. 
See the text following Algorithm (27.24) for a 
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discussion of some steps one might take to 
obtain a usable solution in this situation. 

RNORM If MODE - 1, RNORM is set by the sub¬ 

routine to the value | e | defined by Eq. (27.6). 
This number is computed as 

r ir -1 "ii /2 

U + . o(, - nb+, H 

If MODE - 2 or 3, RNORM is set to zero. 

Example of Usage 

See PROG6 for an example of the usage of BNDACC and BNDSOL. 


LDP 

USER'S GUIDE TO LDP: LEAST DISTANCE PROGRAMMING 

Subroutine Called NNLS, DIFF 
Purpose 

This subroutine computes the solution of the following constrained least 
squares problem: 

Problem LDP: Minimize ||x|| subject to Gx ;> h. Here G is an M x N 
matrix and h is an M-vector. 

No initial feasible vector, x 0 , is required. Thus the subroutine LDP can be 
used to obtain a solution to an arbitrary set of linear inequalities, Gx h. If 
these inequalities have no solution the user is notified by means of a flag 
returned by the subroutine. 

Method 

This subroutine implements Algorithm LDP (23.27), which reduces 
Problem LDP (23.3) to a related Problem NNLS (23.2). Problem NNLS is 
solved with subroutine NNLS and a solution to Problem LDP is then ob¬ 
tained or it is noted that no solution exists. 



268 DESCRIPTION AND USB OF FORTRAN CODES 


AFP. C 


Usage 

DIMENSION G(MDG,n,), H(«,). X(«,). W(n 3 ) 

INTEGER INDEX(m,) 

CALL LDP (G, MDG, M, N. H. X, XNORM. W. INDEX, MODE) 

The dimensioning parameters must satisfy MDG M, /lj ;> N, m, ^ M, 
and n 3 ^ (M + 2)*(N + 1) + 2*M. 

The subroutine parameters are defined as follows: 

G(, ), MDG, M, N The array G( ,) has first dimensioning parameter 

MDG and contains the M x N matrix G of Problem 
LDP. Either M ;> N or M < N is permitted and 
there is no restriction on the rank of G. The contents 
of G( ,) are not modified. 

H( ) The array H( ) contains the M-vector h of Problem 

LDP. The contents of H( ) are not modified. 

X( ) The contents of the array X( ) do not need to be 

defined initially. On normal termination (MODE = 1) 
X( ) contains the solution vector St. On an error 
termination (MODE ^ 2) the contents of X( ) are 
set to zero. 

XNORM On normal termination (MODE = 1) XNORM con¬ 

tains !1 J2| j. On an error termination (MODE ^ 2) 
the value of XNORM is set to zero. 

W( ) This array is used as temporary working storage. 

INDEX( ) This array is used as temporary type INTEGER 

working storage. 

MODE This flag is set by the subroutine to indicate the 

status of the computation on completion. Its value 
indicates the reason for terminating: 

1 = successful. 

2 = bad dimension. The condition N <, 0 was noted 

in subroutine LDP. 

3 = maximum number (3*M) of iterations 

exceeded in subroutine NNLS. 

4 = inequality constraints Gx^>h are incompatible. 

Example 1 

See program PROGS for one example of the usage of this subroutine. 
Example 2 

Suppose a 5 x 15 matrix G and a S-dimensional vector h are given. The 
following code will solve Problem LDP for this data: 
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DIMENSION G(6,16), H(6), X(15), W(122) 

INTEGER INDEX (6) 

DO 10 1*1,5 
H(l) - h, 

DO 10 J -1.15 
10 G(l. J) = g„ 

CALL LDP (G, 6,5, IS, H, X, XNORM, W, INDEX. MODE) 
GO TO (20.30,40,60), MODE 

20 [Successful return. The solution vector X is in X( ).] 

30 [Bad dimension. N ^ 0 was noted.] 

40 [Excessive iterations (more than 3*M) in subroutine NNLS.] 

SO [The inequalities Gx^h are incompatible.] 


NNLS 

USER'S GUIDE TO NNLS: NONNEGATIVE UNEAR 

LEAST SQUARES 

Subroutines Called HI 2. G1, G2, DIFF 
Purpose 

This subroutine computes a solution vector, X, for the following con* 
strained least squares problem: 

Problem NNLS: Solve Ax subject to x ;> 0. 

Here A is a given M x N matrix and b is a given m-vector. This problem 
always has a solution but it is nonunique if the rank of A is less than N. 

Method 

Problem NNLS is solved using Algorithm NNLS (23.10). It can be proved 
that this algorithm converges in a finite number of iterations. The technique 
for adding and removing vectors in the QR decomposition of A is that 
described in Method 3 of Chapter 24. 

Usage 


DIMENSION A(MDA, n,), B(m,), X(n,), W(»,), Z(m,) 
INTEGER INDEX(n,) 

CALL NNLS (A, MDA. M, N, B. X. RNORM.W, Z, INDEX, MODE) 
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The dimensioning parameters must satisfy MDA;> M, n, ;> N, w, ;> M. 
The subroutine parameters are defined as follows: 


A(,), MDA, M, N 


B( ) 

X( ) 

RNORM 

W( ),Z< ) 

INDEX( ) 
MODE 


The array A(,) has first dimensioning parameter 
MDA and initially contains the M x N matrix A. 
Either M ^ N or M < N is permissible. There is no 
restriction on the rank of A. On termination A(, ) 
contains X= QA, where Q is an M x M orthogonal 
matrix implicitly generated by this subroutine. 

The array B() initially contains the M-vector, b. On 
termination B( ) contains Qb. 

The initial content of the array X( ) is arbitrary. On 
termination with MODE = 1 X( ) contains a solu¬ 
tion vector, Jg^O. 

On termination RNORM contains the euclidean 
norm of the final residual vector RNORM 
\\b-A*\\. 

The initial contents of these two arrays are arbitrary. 
On termination W( ) will contain the N-dimensional 
dual vector w = A T (b — AX). The array Z( ) is work¬ 
ing space. 

The initial content of this array is arbitrary. The array 
INDEX( ) is integer working space. 

This flag is set by the subroutine to indicate the status 
of the computation on completion. Its value indicates 
the reason for terminating: 

1 = successful. 

2 = bad dimensions. One of the conditions M <, 0 

or N <, 0 has occurred. 

3 = maximum number (3*N) of iterations has been 

exceeded. 


Error Message 

The subroutine will write the error message 


NNLS QUITTING ON ITERATION COUNT 


and exit with a feasible (approximate solution), X, if the iteration count 
exceeds 3*N. 

Example of Usage 

Solve the least squares problem Ax~b subject to x ^ 0 where A is a 20 
x 10 matrix and b is a 20-dimensional vector. 
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DIMENSION A(20,10), B(20), X(10), W(10), Z(20), INDEX(IO) 
DO 10 I - 1. 20 
B(t) = b, 

DO 10 J = 1,10 
10 A(l, J) = a„ 

CALL NNLS (A, 20,20,10, B, X, RNORM, W, Z. INDEX, MODE) 


H12 

USER'S GUIDE TO H12: CONSTRUCTION AND APPLICATION 
OF A HOUSEHOLDER TRANSFORMATION 

Purpose 

This subroutine implements Algorithms HI and H2 (10.22). Given an m- 
vector v and integers /, and l u this subroutine computes an m-vector u and 
a number s such that them x « (Householder) symmetric orthogonal matrix 
Q — I + (uu^/isu,,) satisfies Qv = w where w, = v, for t < l„ | w, r | = 
(«,* + v?) 1/J , w, = v, for /, < i < /„ and w, = 0 for l x <,i<, m. Op¬ 

tionally this matrix Q may be multiplied times a given set of NCV w-vectors. 

Usage 

CALL H12(MODE. LPIVOT, LI, M, U, IUE. UP, C, ICE,ICV, NCV) 

The parameters are defined as follows: 

MODE Set by the user to the value 1 or 2. If MODE =* 1, 

the subroutine executes Algorithm HI (10.22), 
which computes a Householder transformation 
and, if NCV > 0, multiplies it times the set of 
NCV m-vectors stored in the array C. If MODE - 
2, the subroutine executes Algorithm H2 (10.22), 
which assumes that a Householder transformation 
has already been defined by a previous call with 
MODE = 1 (Algorithm HI) and, if NCV > 0, 
multiplies it times the set of NCV m-vectors stored 
in the array C. 

LPIVOT, LI, M If these integers satisfy 


1 ^ LPIVOT < LI <; M 
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then they constitute the quantities /„ l u and m 
defined above under Purpose. If these inequalities 
are not all satisfied, then the subroutine returns 
without doing any computation. This implicitly 
effects an identity transformation. 

U( ), IUE, UP The array U( J contains M elements with a positive 

storage increment of IUE between elements. If 
MODE = 1, the array U( ) must initially contain 
the m-vector v stored as U(1 + (J — 1)*IUE) — 
«,,./ = 1,.... m. The subroutine will compute s 
and Uj, (see Purpose), storing s in place of v,, and 
storingu ; , in UP. The other elements of U( ) remain 
unchanged; however, the elements v, for /, <Lj <, 
m will be regarded on exit as constituting elements 
Uj since Uj = v } for U ^ m - If MODE ■ 2, 
the contents of U( ) and UP must contain the 
results produced by a previous call to the subrou- 
tine with MODE * 1. 

Example 1: If v is stored in a single subscripted 
Fortran array called W with W(l) :=* v„ then the 
parameters "U, IUE" should be written as "W, 1". 

Example 2: If v is stored as the Jth column of 
a doubly subscripted Fortran array called A with 
A(l, J):= v„ then the parameters "U,IUE" should 
be written as "A(1, J), 1". 

Example 3: If v is stored as the Ith row of a 
doubly subscripted Fortran array called A with 
A(l, J) := v t and A is dimensioned A(50,40), 
then the parameters "U, IUE" should be written 
as "A(l, 1). 60". 

C( ), ICE, ICV, NCV If NCV ^ 0, no action is taken involving the 

parameters C( ), ICE, and ICV. If NCV > 0, 
the array C( ) must initially contain a set of NCV 
m-vectors stored with a positive storage increment 
of ICE between elements of a vector and a storage 
increment of ICV between vectors. Thus if z„ 
denotes the ith element of the/th vector, then C(1 + 
(i — 1)*ICE + (j — 1)*ICV) = z tJ . On output, 
C() contains the set of NCV m-vectors resulting 
from multiplying the given vectors by Q. Hie fol¬ 
lowing examples illustrate the two most typical 
ways in which the parameters C, ICE, ICV, and 
NCV would be used. 
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Example 1: Compute E~ QF where F is a SO 
x 30 matrix stored in a Fortran array F dimen¬ 
sioned as F(60,40). Here M = 50 and the param¬ 
eters "C, ICE, ICV, NCV" should be written as 
"F, 1,60,30". 

Example 2: Compute G = FQ where F is a 20 
x 60 matrix stored in a Fortran array F dimen¬ 
sioned as F(30,70). Here M ■ 00 and the param¬ 
eters "C, ICE, ICV, NCV" should be written as 
"F, 30.1.20". 


Usage Examples 

A. Reduce an n x n real matrix A to upper Hessenberg form using House¬ 
holder transformations. This is often used as a first step in computing 
eigenvalues of an unsymmetric matrix. 


QA&=*H 
h u h i2 
fin 





DIMENSION A(60, 50). UP(60) 

IF(N.LE.2) GO TO 20 
DO 10 I - 3. N 

CALL H12(1,I-1,I.N,A(1,I-2),1,UP(I-2),A(1.I-1 ),1,50,N-l+2) 
10 CALL HI2(2,1-1,1, N, A(1,1-2), 1, UP(l-2), A, 60,1, N) 

20 CONTINUE 


Note that the matrix H occupies the upper triangular and first subdiago¬ 
nal part of the array named A on output. The data defining Q occupies 
the remaining part of the array A and the array UP. 

B. Suppose that we have the least squares problem Ax = b with the special 
form 



^JIX* 


rfgxi 

A = 


. b = 

*1X1 


_iw 


-2*xi . 


where R is upper triangular. Use Householder transformations to reduce 
this system to upper triangular form. 


274 DESCRIPTION AND TOE OF FORTRAN CODES 


AFP. C 


The augmented matrix [A:b] occupies the (N + K + 1) x (N + 1) 
-array named A. This is an example of block by block sequential accu¬ 
mulation as described in Algorithm SEQHT (27.10). 

DIMENSION A(50, 30) 

NP1 ** N +1 
DO10J«1,NP1 

10 CALL H12(1,J,N+2,N+K+1,A(1,J),1,T,A(1,J+1),1,60,N+1 -J) 

Notice that this code does not explicitly zero the lower triangular part of 
the A-array in storage. 


G1, G2 

USER'S GUIDE TO G1 AND G2: CONSTRUCTION AND 
APPLICATION OF ROTATION MATRICES 


Purpose 

Given the numbers jc, and x lt the subroutine G1 computes data that 
defines the 2 x 2 orthogonal rotation matrix G such that 

■h-u :)[a=rv !) i=[;] 

The subroutine 62 performs the matrix-vector product 



See Algorithms G1 (10.25) and G2 (10.26) for further details. 

Input 

(a) The numbers x, and x z are input to G1. 

(b) The numbers c, s, z u and z z are input to G2. 

Output 

(a) The numbers c, s, and r are output by G1. 

(b) The matrix product [g 1 ~\ is output by G2 following a previous execution 
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of G1 that defined the appropriate c and s of the G matrix. The output 
quantities 2, and f 2 replace the input quantities z, and z 2 in storage. 

Usage 

CALL G1(X1,X2,C,S,R) 

CALL G2(C,S,Z1,Z2) 

Usage Example 

Suppose that we have the least squares problem Ax^b with the special 
form 



&n** 


4x1 

A =* 


, 6 = 

*|X! 


.y i 


.*Ixl . 


where R is upper triangular. 

Use rotation matrices to reduce this system to upper triangular form and 
reduce the right side so that only its first n + 1 components are nonzero. 
The augmented matrix [A: b] occupies the (N + 2) x (N + 1 )-array named 

A. 

This is an example of a row by row “sequential accumulator.” See the 
remarks following Algorithm SEQHT (27.10). 

hi PI - N + 1 
DO 10 I -1, NP1 

CALL G1 (A(l, I), A(N+2,1), C, 8, A(l, I)) 

A(N + 2.1) - 0. 

IF (I.GT.N) GO TO 20 
DO 10 J * I. N 

10 CALL G2 (C,S,A(I.J+1).A{N+2.J+1)) 

20 CONTINUE 


MFEOUT 

USER’S GUIDE TO MFEOUT: MATRIX OUTPUT 

SUBROUTINE 


Purpose 

This subroutine outputs a formatted representation of an M 
x N matrix with one of two built-in titles, row and column 
numbering, and row labels. It is designed to be called by the 
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singular value analysis subroutine SVA to print the V-matrix 
and the matrix of candidate solution vectors. 

The matrix will be printed by blocks of columns. Each block 
will contain all M rows of the matrix and as many columns as 
will fit within the character width specified by WIDTH. A line 
of output will contain a Fortran “carriage control” character, a 
row number, a row label taken from NAMES( ), and elements 
of the matrix occupying 14 character positions each. 

Usage 

INTEGER MDA, M, N, MODE, UNIT, WIDTH 
REAL A(MDA,m) 

CHARACTER NAMES(n x ) * ( lennam ) 

CALL MFEOUT(A, MDA, M, N, NAMES, MODE, UNIT, 
WIDTH) 

The dimensioning parameters must satisfy MDA > M, n x > 
N, and lennam > 1. 

The subroutine parameters are defined as follows: 



Array containing an M x N matrix to be output. 

MDA 

Dimension of first subscript of the array A(,). 


M,N 


Number of rows and columns, respectively, in the matrix to be 
output. 

NAMES( ) 

NAMES(i) contains a character string label to be output in as¬ 
sociation with row i of the matrix. If NAMES(l) contains only 
blank characters, it will be assumed that no names have been 
provided, and the subroutine will not access the NAMES( ) 
array beyond the first element and will output blank labels for 
all rows. 
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MODE 

MODE = 1 provides headings appropriate for the V matrix of 
the singular value decomposition and uses a numeric format of 

4pfl4.0. 

MODE = 2 provides headings appropriate for the array of 
candidate solutions resulting from singular value analysis and 
uses a numeric format of gl4.6. 

UNIT 

Selects the output unit. If UNIT > 0, then UNIT is the 
output unit number. If UNIT = —1, output will go to the 
unit, i.e., the standard system output unit. 

WIDTH 

Selects the width of output lines. Each output line from this 
subroutine will have at most max(26,min(124,WIDTH)) char¬ 
acters plus one additional leading character for Fortran “car¬ 
riage control,” which will always be a blank. 


GEN 

USER'S OUIDE TO GEN: DATA GENERATION FUNCTION 

Purpose 

This FUNCTION is used by PROG1, PROG2, and PROG3 to generate 
data for test cases. By basing the generation method on small integers it is 
intended that the same test cases can be generated on virtually all computers. 

Method 

This FUNCTION generates two sequences of integers: 

/„ = 5 

7* = (891#/*_,) mod (1000) * = 1,2,... 

and 

/. = 7 

J, = (457*/,. ,) mod (997) /= 1,2,... 

The sequence {/*} has period 10, while the sequence {/,} has period 332. 
On the *th call after initialization, GEN produces the REAL output value 
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FLOAT (/* - 500) + ANOISE*FLOAT (J, - 498) 

The next member of the sequence {/,} is produced only when ANOI8E > 0. 
No claim is made that this sequence has any particular pseudorandom 
properties. 

Usage 

This FUNCTION must be initialized by a statement of the form 

X - GEN(A) 

where A < 0. In this case the value assigned to X is zero. 

The next number in the sequence defined above is produced by a statement 
of the form 

X - QEN(ANOISE) 

where the input parameter ANOISE is nonnegative. 


DIFF 

USER'S GUIDE TO DIFF 


Purpose 

This FUNCTION is used by the subroutines HFTI, QRBD, LDP and 
NNL8 to make the test “If ((x -f A) — x) ^ 0” which is used in place of 
the test “If (| A | > 9 1 x D M where if is the relative precision of the machine 
floating point arithmetic. 

Method 

In the intended usage of this FUNCTION the intermediate sum z — x 
+ A is computed in the calling program using imprecision arithmetic. Then 
the difference d = z — x is computed by this FUNCTION using imprecision 
arithmetic. 

Usage 

This FUNCTION can be used for the test described in Purpose with 
an arithmetic IF statement of the form 

IF(DIFF(X+H, X)) 10,20,10 

The statement numbered 10 corresponds to the condition |A| > if |x|. The 
statement numbered 20 corresponds to the condition |Aj <; if |x|. 


BVLS 

USER’S GUIDE TO BVLS: BOUNDED VARIABLES LEAST 

SQUARES 


Purpose 

This Fortran 90 subroutine finds the least squares solution to an M x N 
system of linear equations Ax ~ b, subject to the constraints 

(*i<Xi< & i =s 1,..., N 

It is permitted to have M > N or M < N. This problem always has a 
solution, but it is generally nonunique if the rank of A is less than N . 


Method 

Problem BVLS is solved using an algorithm of C. Lawson and R. Han¬ 
son. It is a generalization of the algorithm NNLS described in Chapter 
23. In the following descriptions we refer to sets T, V, and Z. Set T con¬ 
sists of variables fixed throughout the solution process due to their lower 
and upper bounds being equal. All other variables are in sets V or Z. 
The membership of sets V and Z may change during the solution process. 
Variables in Set V are determined to minimize the objective FUNCTION 
subject to the variables in Set Z being held (temporarily) at fixed values. 
A variable in Set Z may be at one of its bounds or at zero. 


Usage 

CALL BVLS(A, B, BND, X, RNORM, NSETP, W, INDEX, IERR) 


The subroutine parameters are defined with the following in¬ 
terface: 


interface 

subroutine bvls(A, B, BND, X, RNORM, & 

NSETP, W, INDEX, IERR) 

real(kind(leO)) AB(:), BNDX(:) f RNORM, W(:) 
integer NSETP ; INDEXf:), IERR 
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end subroutine 
end interface 


ApNTENT(inout)] 

On entry, A(:,:) contains the M x N matrix A . On return, 
A(:,:) contains the product matrix QA, where Q is an M x M 
orthogonal matrix generated implicitly by this subroutine. The 
values M and N are the respective number of rows and columns 
in A(:,:). Thus M=size(A,l) and JV=siRe(A,2). Required 
are M > 0 and N > 0. 


B(:) [INTENT(inout)] 

On entry, B(:) contains the M-vector, b. On return, B(s) 
contains Qb. 


BND(1:2,:) pNTENT(in)] 

The lower bound for Xi must be given in BND(l,i) and the 
upper bound ft in BND(2,i). Required are (Xi < ft. To indi¬ 
cate Xi has no lower bound, set BND(1,«) = — HUGE(X.OeO). 
To indicate that Xi has no upper bound, set BND(2,t) = 
HUGE(l.OeO). 

X(s) [INTENT(out)] 

On entry, X(:) need not be initialized. On a normal return, 
X(:) will contain the solution iV-vector. 


RNORM (INTENT(out)] 

On a normal return this is the Euclidean norm of the residual 
vector. 
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NSETP [INTENT(out)] 
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Indicates the number of components of the solution vector, 
x<0, that are in Set V. NSETP and INDEX(:) together 
define Sets V, Z, and T. 


W(:) [INTENT(out)] 

On return, W(:) will contain the dual vector w. This is the 
negative gradient vector for the objective FUNCTION at the 
final solution point. For j € T, W (j) may have an arbitrary 
value. For j 6 V, W(j) =0. For j 6 Z, W(j) is < 0, > 0, or 0, 
depending on whether X(j) is at its lower bound, at its upper 
bound, or at zero with zero not being a bound. 


INDEX(:) pNTENT(out)] 

An INTEGER array of length at least N. On exit the value of 
NSETP and the contents of this array define the sets V, Z, and 
T. The indices in INDEX(1: NSETP) identify V. Letting nf 
denote the number of pairs of lower and upper bounds that are 
identical, the indices in INDEX(NSETP+1: N-nf) identify 
Z, and the indices in !NDEX(iV'-n/4T: N) identify T. 


IERR [INTENT(out)] 


Indicates the status on return. 


= 0 Solution completed OK. 
= lAf<0orlV<0 


= 2 One array has inconsistent size. 

Required are size(B) > M, size(BND,l) = 2, size(BND,2) 
> N , size(X) > N, size(W) > N, and size(INDEX) > N. 

— 3 Input bounds are inconsistent. Required are BND(l,i) < 
BND(2,t), t=l,... ,N. 

— 4 Exceeds maximum number of iterations, ITMAX = 3 *N. 


This last condition may be caused by the problem being very ill- 
conditioned. It is possible, but uncommon, that the algorithm 
needs more iterations to complete the computation. The itera¬ 
tion counter, ITER, is increased by one each time a variable is 


moved from Set Z to Set V. 
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Remark 

It is not required that A be of full column rank. In particular it is 
allowed to have M < N. In such cases there may be infinitely many 
vectors x that produce the (unique) minimum value of the residual vector 
length. The one returned by this subroutine does not have any particular 
distinguishing properties. It does not necessarily have the least possible 
number of nonzero components nor the minimum euclidean norm of all 
solutions. 


Functional Description 


The algorithm used is a generalization of Algorithm NNLS, given in 
Chapter 23, for the solution of the least squares problem with nonnegativity 
constraints on the variables. 

Given A and 6, for any vector x we may define the residual vector, 
r = 4- Ax, and the dual vector w = A T r. The vector w is half the 
negative gradient vector of the objective function ||r|| . Thus, for a vector 
x that does not give the minimum of this objective FUNCTION, w points 
in the downhill direction. In particular for the unconstrained problem, a 
vector x is the solution if and only if its dual vector w is zero. 

In the bounded variables problem the solution is characterized by the 
signs of components of w being such that for each i, either w* = 0 or a 
perturbation of Xi in the direction Wi is blocked by one of the given bounds 
for x^ Thus x is a solution to the bounded variables problem if and only 
if for each i one of the following is true: 


1) a* = & and x» = a* 2) Xi = a* and Wi < 0 
3) Xi = Pi and wi > 0 4) a* < x< < Pi and Wi = 0 


Every variable is first set to a value as close to zero as possible, subject 
to satisfying its bounds. During execution of the algorithm the variables 
are classified into three disjoint sets, called V, and Z. Set !F consists of 
the variables that are permanently fixed at a constant value due to their 
upper and lower bounds being equal. 

Initially all variables not in Set T are placed in Set Z. As the computa¬ 
tion proceeds, the variables not in Set T will move between Sets Z and V. 
At each stage of the procedure Set Z will consist of variables that are tem¬ 
porarily being held at fixed values, which may be a lower or upper bound or 
zero. Set V will consist of variables whose values are determined by solving 
the least squares problem subject to the variables in Sets T and Z having 
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fixed values. The augmented matrix [A : b] will be successively modified by 
the application of orthogonal transformations from the left determined so 
that the columns associated with Set V form an upper triangular matrix. 

A variable will not be moved into Set V if its associated column in 
A is nearly linearly dependent on the columns already in Set V. This 
test uses the machine precision obtained from the elemental Fortran 90 
function EPSILON(l.OeO). It is possible for this rejection of a variable 
to cause the algorithm not to achieve the minimum value for the objective 
FUNCTION. The criteria and methods for moving variables between 
Sets Z and V are similar to those described for algorithm NNLS but with 
changes to handle two-sided bounds. 

On return, NSETP is the number of variables in Set P, and the indices 
of these variables are given in INDEX(t)* % = l v ..NSETP. These 
variables will generally not be at constraint boundaries, whereas the other 
variables, INDEX(i), i = NSETP+1, ..., N, will be at their lower or 
upper bounds or (exceptionally) at the value zero. 



APPENDIX 

D DEVELOPMENTS 

FROM 1974 TO 1995 

Introduction 

In this appendix we remark on developments since 1974 directly related to 
the topics treated in the main body of the book, plus a few topics that were 
not previously discussed. The remarks are grouped by the book chapters 
and appendices to which they relate and by the names of the new topics. 
Most of the citations in this appendix are to the references at the end of 
this appendix. In the few that refer to the original bibliography the year 
is prefixed with “b-.” 

Books and Monographs 

The monograph Least Squares Methods by Bjorck (1990), thoroughly de¬ 
scribes methods of least squares computation. A_significantly expanded 
revision of this monograph is projected for publication as a book in 1995: 
Numerical Methods for Least Squares Problems . Ake Bjorck maintains 
an extensive bibliography on least squares computation in the directory 
pub/references at the Internet address math.liu.se. Information there is 
available by anonymous ftp. 

Matrix Computations by Golub and Van Loan (1989) gives a compre¬ 
hensive overview of contemporary methods of computational linear algebra, 
with special attention to block-oriented algorithms that perform efficiently 
on computers that have a high-speed cache memory between the processor 
and the lower-speed main memory. 

The LAPACK library of linear algebra software, LA PACK Users * Guide¬ 
book by Anderson, et al. (1995) provides Fortran implementations of algo¬ 
rithms for linear least squares, singular value decomposition, linear equa¬ 
tions, and eigensystem problems for dense matrices. The authors of LA- 
PACK developed new block-oriented algorithms and developed new accu¬ 
racy analyses. 

Theory of the Combination of Observations Least Subject to Errors by 
Stewart (1995) is of historic and scholarly significance, being a translation 
from Latin and German into English of classic papers on least squares and 
related topics by Carl Friedrich Gauss. 

Other books and monographs that have substantial coverage of topics 
that have applications to least squares computation are Nonlinear Parame¬ 
ter Estimation by Bard (1974), Factorization Methods for Discrete Sequen¬ 
tial Estimation by Bierman (1977), Large Sparse Numerical Optimization 
by Coleman (1984), Handbook for Matrix Computations by Coleman and 
Van Loan (1988), Numerical Linear Algebra and Applications by Datta 
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(1995), Numerical Methods for Unconstrained Optimization and Nonlinear 
Equations by Dennis and Schnabel (1983), Direct Methods for Sparse Ma¬ 
trices by Duff, et al. (1987), Practical Optimization by Gill, et al. (1981), 
Optimization Software Guide by More and Wright (1993), Direct Methods 
for Sparse Matrices by 0sterby and Zlatev (1983), The Symmetric Eigen¬ 
value Problem by Parlett (1980), Linear Algebra and Its Applications by 
Strang (1988), The Total Least Squares Problem—Computational Aspects 
and Analysis by Van Huffel and Vandewalle (1991), and Spline Methods for 
Observational Data by Wahba (1990). 

Chapters 1 and 2. Statement of least squares problems 

In the usual least squares problem, Ax & 6, one regards b as containing 
observational error and A as not having observational error. Problems in 
which there is observational error in A also arise. Such problems have been 
discussed in the statistical literature under the names orthogonal regression 
and errors in variables . The name total least-squares was introduced for 
this problem class in Golub and Van Loan (1980), where systematic solu¬ 
tion methods using the singular value decomposition are described. This 
problem class is treated at length in the book by Van Huffel and Vandewalle 
(1991). Additional research on the total least squares problem is reported 
in Boggs, et al. (1987), Aran (1992), De Moor (1993b), Fierro and Bunch 
(1994), Fierro, et al. (1993), Reilly, et al. (1993), Rosen, et al. (1996), Ten 
Vregelaar (1995), Van Huffel (1992), and Wei (1992c). 

Chapters 3 and 4. Orthogonal decompositions 

Chapters 3 and 4 present, respectively, the orthogonal decomposition A = 
HRK t and the singular value decomposition A = USV T . Here H, K , U, 
and V are orthogonal, R is triangular, and S is diagonal. The HRK T de¬ 
composition includes the very important QR decomposition as the special 
case of K = I. 

Two other orthogonal decompositions that have been studied are the CS 
and GSVD (generalized SVD) decompositions. The relation of the GSVD 
to the SVD is analogous to the relation between the eigensystem problem 
for a single matrix Ax —x A = 0 and the eigensystem problem for a matrix 
pencil Ax — BxX = 0. The CS decomposition has been used mainly as an 
aid in studying and computing the GSVD. 

The GSVD was introduced and studied in Van Loan’s thesis (1973). 
Exposition and references on the CS and GSVD decompositions appear 
in Golub and Van Loan (1989) and Bjorck (1990, 1995). The GSVD is 
further treated in Bai and Demmel (1991), Bai and Zha (1993), Hansen 



286 DEVELOPMENTS FROM 1974 TO 1995 




(1990b), Paige (1984,1986), Paige and Saunders (1981), Park (1994), Stew¬ 
art (1982), Sun (1983), and Van Loan (1976, 1985). Generalizations of the 
QR decomposition are treated in Paige (1990). 

Chapters 5, 8, 9, 15, 16, and 17. Perturbation bounds 
and computational error analyses for QR, Cholesky, and 
singular value decompositions and the solution of the 
least squares problem 

Results given in Chapter 5 show that small absolute perturbations to the 
elements of a matrix, A> can give rise to only small absolute perturbations 
of its singular values. This allows for the possibility that small perturba¬ 
tions of a matrix may give rise to large relative perturbations of the small 
singular values. Studies have identified conditions under which small rela¬ 
tive perturbations of small singular values can be guaranteed. In LAPACK, 
algorithms that obtain good relative accuracy for small singular values are 
used when the structure of the matrix permits this. See Demmel and Ka¬ 
lian (1990) and other references in Anderson, et al. (1995). Perturbation 
properties of singular values are also studied in Vaccaro (1994). 

Perturbation analyses for the QR decomposition are the subject of Sun 
(1995) and Zha (1993). The latter paper gives component-wise bounds. 
Error analysis for an algorithm for the equality constrained least squares 
problem is given in Barlow (1988). Perturbation and error analyses for the 
Cholesky and LDL T factorizations are given in Sun (1992). Perturbation 
bounds for the solution of the least squares problem are given in Ding and 
Huang (1994), Stewart (1977), and Wei (1990). 

Chapter 6. Bounds for the condition number of a tri¬ 
angular matrix 

Computing the singular values of a matrix generally provides the most 
reliable way to compute its condition number. There has been a continuing 
search for less expensive ways of estimating the condition number or rank 
of a matrix that still have a reasonable level of reliability. It is also desirable 
that such methods integrate conveniently with the main problem-solving 
algorithm, which may involve updating, sparse matrix techniques, and so 
on. 

The methods discussed in Chapter 6, i.e., inspection of diagonal ele¬ 
ments after Householder triangulation using column interchanges, are in¬ 
expensive but not as reliable as one might wish. A method of condition 
estimation introduced in LINPACK [Dongarra, et al. (1979)] brought in¬ 
creased attention to this class of algorithms. Chan (1987) contributed 
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new algorithmic ideas for rank estimation and introduced the term rank- 
revealing decomposition, which has been further treated by a number of 
authors, e.g., Barlow and Vemulapati (1992b), Bischof and Hansen (1991, 
1992), Chan and Hansen (1990, 1992, 1994), Chandrasekaran and Ipsen 
(1994), Fierro and Hansen (1993), Higham (1987), Mathias (1995), Park 
and Eld£n (1995a), Shroff and Bischof (1992), and Van Huffel and Zha 
(1993). 

Chapter 10. Algorithms for application of orthogonal 
transformations 

Reorganization of the Householder transformation into a block form is 
treated in Dietrich (1976), Dongarra, et al. (1986), Kaufman (1987), 
Schreiber and Parlett (1988), and Schreiber and Van Loan (1989). The 
WY algorithm is summarized in Golub and Van Loan (1989). 

An alternative realization of the 2-multiply Givens algorithm that is 
intended to have advantages for implementation on a systolic array is given 
in Barlow and Ipsen (1988). Another formulation of the 2-multiply Givens 
algorithm, which reduces the number of inner loop operations in some 
compiler/processor combinations, is given in Anda and Park (1994). 

Chapter 12. Computation of the covariance matrix with 
constraints 

The covariance matrix computation described in Chapter 12 can easily be 
adapted to the case of Problem LSE (Least Squares with Equality Con¬ 
straints) which is treated in Chapters 20-22. The LSE problem is stated 
in Eqs. (20.1)-(20.3) and one solution method for the case of Rank (C) = 
m\ is presented in Eqs. (20.19)-(20.24). 

Define 

5 = (El E 2 ) 1 

where E 2 is defined by Eq. (20.19). 5 is the unsealed covariance matrix 
for y 2 which is determined from Eq. (20.22). 

Since y\ of Eq. (20.20) is known with certainty, the unsealed covariance 
matrix for y is 

t/ = 

and the scaled covariance matrix for x is 

T = o 2 KUK t 

0.2 __ T _ _ T 

m 2 — (n — mi) mi + m 2 — » 
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If the least squares problem of Eq. (20.22) is solved using a QR factor¬ 
ization of Ev, one will have 



where Q is orthogonal 1712x7712 and R is upper triangular (n-mi)x (n-mj). 
Then S can be computed as 

5 - R-\R- l ) T 

Deriving formulas for computation of the covariance matrix for use with 
the solution method of Chapter 21 is only slightly more involved. Using 
the quantities defined in Eqs. (21.4)-(21.7), the resulting formulas are: 

W — (i?f Ea) -1 

H = Ci l C 2 

r = a 2 W[ -H T I ] 

where a is defined as above. Since the expression defining H occurs as 
a subexpression in Eq. (21.4), H can be obtained as a byproduct of the 
solution algorithm. 

Chapter 13. The underdetermined full rank problem 

The problem Ax = b where A is an m x n matrix with m < n and 
Rank(A) = m will have an (n — m)-dimensional linear flat of solutions. 
In Chapter 13 we gave a method for finding the unique minimum length 
solution making use of orthogonal transformations of A from the right. 
Paige (1979a, 1979b) treats the situation in which one wants only some 
portion of the x-vector to have minimum length. See also the heading 
“Generalized Least Squares” in Golub and Van Loan (1989). 

For k < m, let x\ denote the first k and X2 denote the last n — fe 
components of x. Let A be partitioned similarly: A — [A\ : A2). Suppose 
one wants a solution of Ax = 6 with ||x 2 || minimized. 

Apply orthogonal transformations to the partitioned matrix [Ax: A 2 : 6] 
from the left to transform A\ to upper triangular form: 
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The dimensions of RuS\ and S 2 are ixfc,kx(n-i), and (m — k) x 
(n — k) . Assume 

Rank(Sa) — m-k 

Compute an orthogonal matrix K that triangularizes S 2 from the right: 

S 2 K * [ r 2 : 0 ] 

The dimension of T 2 is (m — k) x (m — k) . Introduce the change of variables 

x 2 = Ky = [K, : K 2 ] [ £ ] 

The matrices Ai and /f 2 have dimensions (n — k) x (m — k) and (n — fc) x 
(n — m) and the vectors 3/1 and 1/2 have m — k and n — ra components. 
Writing 

Si/r « [T x : C/xJ 

the problem becomes 


R\X\ + Tipi + Uxjft = Pi 
lalfi = Pfc 

Any value can be assigned to y 2 . However, 2/2 = 0 gives the minimum value 
for ||xa||. With y 2 = 0, compute y\,xu and x 2y respectively, from 

T22/1 = 92 

RiXx = pi — Tipi 

= ATiyi 

In the referenced papers, Paige applied this formulation to the classical 
weighted least squares problem. Suppose the observation vector b in the 
system Az & b is affected by errors assumed to have mean zero and 
variance-covariance matrix <r 2 C. Here C is symmetric, nonnegative def¬ 
inite, with Cholesky factorization C = LL T . 

If C is positive definite and thus L is nonsingular, the appropriate 
weighted least squares problem is that of finding z to minimize \\L~ l (Az - 
6 )||. Writing r = L~ l (Az — 6 ), the problem becomes one of finding z and r 
to minimize j|r|| subject to Az — Lr = b. This is an equivalent statement 
when L is nonsingular and remains well defined even when L is singular. 
This system can be analyzed as above if one identifies A with A\> — L with 
A 2y z with Xu and r with x 2 . 

Sparse underdetermined linear systems are treated in George, et al. 
(1984). 
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Chapter 18. The SVD algorithm 

Besides the GSVD, mentioned above in connection with Chapters 3 and 
4, a number of other variants on the SVD have been defined and studied. 
See Adams, et al. (1994), De Moor and Van Dooren (1992), Fernando and 
Hammarling (1987), Watson (1992), and Zha (1991, 1992). 

In Section 5 of Chapter 18 it is mentioned that to compute the singular 
values and the V matrix of the SVD of an m x n matrix A when ran is 
large and m » n, computer storage and run-time can be reduced by first 
transforming A to an n x n triangular matrix R by sequential processing 
as in Chapter 27 and then applying the SVD or SVA algorithm to R. 
Operation counts for this approach are included in Table 19.1 in Chapter 
19. 

Chan (1982) incorporates a preprocessing phase of Householder trian¬ 
gulation into an SVD code so it will automatically be used when m is 
suitably larger than n. Bau (1994) elaborates this idea, showing that when 
1 < mjn < 2, the operation count is minimized by executing bidiago- 
nalization, as in the usual SVD algorithm, until the aspect ratio of the 
nonbidiagonalized part of the matrix is approximately 2, applying a QR 
factorization to this remaining part, and finally bidiagonalizing the trian¬ 
gular factor. These approaches improve on the execution time savings of 
the approach given in Chapter 18, but do not achieve the saving of storage 
that may be essential when the product mn is very large and m » n. 

Additional direct SVD algorithms are presented and analyzed in Chan- 
drasekaran and Ipsen (1995) and Demmel and Kahan (1990). For iterative 
SVD algorithms see Golub, et al. (1981) and Vogel and Wade (1994). A 
parallel SVD algorithm is given in Jessup and Sorensen (1994). 


Chapter 19. Other methods for least squares problems 

The method of seminormal equations is treated in Bjorck (1987, 1990) and 
summarized under the heading “Normal Equations Versus QR” in Golub 
and Van Loan (1989). Orthogonality properties of the modified Graro- 
Schmidt algorithm are studied in Bjorck and Paige (1992). 

In a surface-fitting application using tensor product splines, Grosse 
(1980) shows that the expected instability from using normal equations 
is, in fact, not encountered. 


Chapters 20, 21, and 22. Least squares with equality 
constraints 

The problem of least squares with equality constraints (LSE) is of interest in 
its own right as well as its being an essential component of most algorithms 
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for more general constrained least squares and constrained optimization 
problems. Problem LSE is treated in Anda and Park (to appear), Barlow 
and Handy (1988), Barlow and Vemulapati (1992a), Eld£n (1980, 1982), 
and Wei (1992a). 


Chapter 23. Least squares with inequality constraints 

The NNLS algorithm first appeared in the 1974 edition of this book. A 
number of readers of the book have informed us that they have used NNLS 
successfully in a variety of applications. NNLS has also been incorporated 
in a number of commercial software packages. 

Since 1974 we developed a BVLS algorithm and code for the Bounded 
Variables Least Squares problem. This is a generalization of the NNLS 
Algorithm in which the constraints on the variables are a* < x% < ft 
rather than the constraints Xi > 0 of NNLS. The structure of the BVLS 
Algorithm is essentially the same as NNLS with additional details to deal 
with the two-sided inequalities. 

At each stage of the solution process in NNLS the variables are classified 
into two classes, which we call Set Z and Set V . Variables in Set V are 
determined to minimize the objective function subject to the variables in 
Set Z being held (temporarily) at fixed values. In NNLS each Xi is initially 
set to zero and classified as being in Set Z. 

In BVLS there is a third class, Set T, consisting of variables that are 
fixed throughout the process due to their lower and upper bounds being 
identical. If the bounds for an Xi are relatively large in magnitude and 
of opposite signs, it could introduce unnecessary cancellation errors to ini¬ 
tialize Xi at one of its bounds and possibly have it end up at an interior 
value of much smaller magnitude. To avoid this potential source of error 
we initialize each to a value closest to zero that is within its bounds. 
If an Xi is not in Set T, it is initially classified as being in Set Z. As a 
consequence, variables in Set Z are not necessarily at their bounds. There 
is no harm in this, but the algorithm must take it into consideration. 

A hierarchy of more general constrained minimization problems can be 
defined by adding general linear inequality or equality constraints on the 
variables or nonlinear constraints. Furthermore the objective function can 
be generalized from a sum of squares of linear functions of the variables to 
a general quadratic function of the variables or to a general differentiable 
nonlinear function of the variables. 

These more general constrained minimization problems are treated in 
Gill, et al. (1981). For additional research on constrained least squares, 
see Cline (1975), Dax (1991), Guler, et al. (1995), Gulliksson (1994,1995), 
Gulliksson and Wedin (1992), Hanson (1986), Hanson and Haskell (1981, 
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1982), James (1992), Stoer (1992), and Wei (1992b). 

Chapters 25 and 26. Practical analysis and regulariza¬ 
tion in applications 

In an ill-conditioned least squares problem there will typically be a set of 
significantly different candidate solution vectors that reduce the objective 
function to an acceptably small value. The term regularization is used to 
describe a variety of methods that can be used to pick a solution from this 
set that is “reasonable” for the intended application. 

Three methods for comparing candidate solutions for an ill-conditioned 
problem are discussed in Chapter 26. The curve in Figure 26.2 has since 
been called an L-curve by P. C. Hansen. He and others have made a number 
of studies of these and related methods of regularization. 

The method of cross validation for choosing a regularization parameter 
in a possibly ill-conditioned least squares problem is presented in Golub, 
Heath, and Wahba (1979). This topic has led to a considerable amount of 
subsequent research. 

Regularization, including L-curve, cross validation, and other approaches, 
is treated in Eld6n (1977,1984a, 1984b, 1984c, 1990), Fierro, et al. (1993), 
Hanke and Hansen (1993), Hansen (1987, 1989a, 1989b, 1990a, 1990b, 
1990c, 1992b, 1994), Hansen and O’Leary (1993), Hansen, et al.(1992), 
and Zha and Hansen (1990). 

Chapter 27. Updating and downdating of rows, sequen¬ 
tial estimation, and consideration of sparsity and struc¬ 
ture 

Introducing or removing the effect of a row in a least squares problem is 
called row updating or row downdating , respectively. Updating and down¬ 
dating of both rows and columns are at the heart of algorithms for con¬ 
strained optimization, e.g., see Gill, et al. (1981). 

Row updating is the key operation in sequential estimation algorithms 
such as are described in Chapter 27. Such algorithms can be used to 
accumulate the effect of observations into the solution process one at a time 
or in batches. One motivation for sequential algorithms is the reduction 
of computer memory capacity needed, while in other cases the purpose 
is to be able to obtain solutions at many intermediate time points as the 
observations are being acquired. This latter type of application is generally 
treated in the engineering literature with the terminology of filtering . 

The algorithm for sequential solution of a banded least squares prob¬ 
lem given in Chapter 27 can be regarded as an example of an algorithm 
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designed to take advantage of a particular case of sparsity and structure. 
Much attention has been given to algorithms for large sparse, and pos¬ 
sibly structured, least squares problems. This includes both direct and 
iterative methods. Algorithms for the solution of large sparse positive defi¬ 
nite systems are often motivated by their application to partial differential 
equations but also have potential application to least squares problems. 

For research on updating and downdating see Bartels and Kaufman 
(1989), Bendtsen, et al. (1995), Bischof, et al. (1993), Bjorck, et al. (1994), 
Eld6n and Park (1994a, 1994b, 1995), Elhay, et al. (1991), Jones and 
Plassman (1995a, 1995b), Moonen, et al. (1992), Olszanskyi, et al. (1994), 
Park and Eld£n (1995a), Park and Van Huffel (1995), Shroff and Bischof 
(1992), Stewart (1992), Xu, et al. (1994), and Yoo and Park (1995). 

A fundamental paper on the solution of large sparse linear least squares 
problems is George and Heath (1980). Methods for a variety of types 
of large, sparse, and in some cases structured least squares problems are 
treated in Bjorck (1984), Coleman (1984), Dax (1993), Golub, et al. (1981), 
Heath (1982), Matstoms (1992, 1994), Ng and Chan (1994), 0sterby and 
Zlatev (1983), Paige and Saunders (1982), Ray (1995), and Van Der Sluis 
and Van Der Vorst (1990). Methods for large sparse linear least squares 
problems are surveyed in Heath (1984). Sparse underdetermined linear 
systems are treated in George, et al. (1984). 

The point of view and terminology of filtering as an approach to sequen¬ 
tial estimation problems springs primarily from Kalman (b-1960). Sequen¬ 
tial estimation is treated from the point of view of statistical regression by 
Duncan and Horn (1972) and as a structured sparse least squares problem, 
to which orthogonal transformations can be effectively applied, by Paige 
and Saunders (1977). Bierman (1977) presents sequential estimation from 
the filtering point of view and makes effective use of orthogonal transfor¬ 
mations for numerical stability. 

Methods for Cholesky factorization of large sparse positive definite sys¬ 
tems are given in Duff, Erisman, and Reid (1987), George, and Liu (1981), 
Gilbert and Schreiber (1992), Joubert and Oppe (1994), Ng and Peyton 
(1993), and Rothberg and Gupta (1994). 

Nonlinear least squares 

The topic of nonlinear least squares is not explicitly treated in the body 
of this book. This problem is treated in Bard (1974), Bjorck (1990, 1995), 
Dennis, et al. (1981), Dennis and Schnabel (1983), Donaldson and Schn¬ 
abel (1987), Gill, et al. (1981), Hanson and Krogh (1992), Heinkenschloss 
(1993), Huschens (1994), More, et al. (1980), Schnabel and Frank (1984), 
and Wedin and Lindstrom (1988). 
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A subproblem that typically occurs in trust region algorithms for non¬ 
linear least squares is the solution of a linear least squares problem subject 
to a norm constraint. This subproblem is essentially the same as the reg¬ 
ularization problem mentioned above in Chapters 25 and 26, and methods 
for one of these problems are generally also applicable to the other. Papers 
addressing these problems are Golub and von Matt (1991), Heinkenschloss 
(1994), and More (1977). 

It is not uncommon for some variables in a nonlinear least squares 
problem to occur linearly. Special treatment can be given to such problems. 
Such problems are called separable nonlinear least squares problems, and 
the solution approach involving differentiation of a pseudoinverse matrix is 
called the variable projection method. Such algorithms are presented and 
analyzed in Golub and Pereyra (b-1973), Kaufman (1975), Kaufman and 
Sylvester (1992), Kaufman, et al. (1994), Krogh (1974), and Lawton and 
Sylvestre (1971). 

Least squares algorithms involving Toeplitz and Hankel 
matrices 

Fast least squares algorithms have been developed for problems involving 
Toeplitz and Hankel matrices. These problems arise in image analysis. See 
Bojanczyk, et al. (1986), Chan, et al. (1993, 1994a, 1994b), Cybenko 
(1987), Park and Elden (1995b), and Ng (1993). 

There are signal processing algorithms that involve centrohermitian cor¬ 
relation matrices. These have relations to Toeplitz matrices that allow the 
development of sparse transformations that can be used reduce computa¬ 
tions required for most applications. See Linebarger, et al. (1994). 

Applications of least squares 

The following papers deal with the use of least squares computations in a 
variety of applications, including signal processing, maritime and aerospace 
navigation, curve and surface fitting (using spline, radial, and Haar basis 
functions), solution of integral equations, Markov chains, tomography, inte¬ 
ger data, and so on: Arun (1992), Brezinski and Matos (1993), Dax (1992, 
1994), Demetriou (1995), De Moor (1993a), Dowling, et al. (1994), Elhay, 
et al. (1991)* Elliott (1993), Fausett and Fulton (1994), Gander, et al. 
(1994), Golub and Meyer (1986), Goodman, et al. (1995), Grosse (1980), 
Hansen (1992a), Hansen and Christiansen (1985), Hansen, et al. (1992), 
Hanson and Phillips (1975, 1978), Hanson and Norris (1981), Helmke and 
Shayman (1995), Kaufman (1993), Kaufman and Neumaier (1994), Lawson 
(1977, 1984), Linebarger, et al. (1994), Liu (1994), Mason, et al. (1993), 
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Poor (1992), Quak, et al. (1993), Reichei (1991), Sevy (1995), Springer 
(1986), Stewart (1992), Wahba (1990), Williams and Kalogiratou (1993), 
Wood (1994), and Xu, et al. (1994). Data assimilation in the fields of me¬ 
teorology and oceanography gives rise to very large and complex nonlinear 
least squares problems, e.g., see Navon, et al. (1992), Zou, et al. (1993), 
and Zou and Navon (1994). 

Appendix A. Basic Linear Algebra Including Projec¬ 
tions 

For more detailed exposition of the fundamentals of linear algebra, see 
textbooks on the subject, e.g., Strang (1988). 

Appendix B. Proof of Global Quadratic Convergence of 
the QR Algorithm 

The convergence proof presented in Appendix B is due to Wilkinson (b- 
1968a, b-1968b). This proof, which is based on analysis of the history of 
individual scalar components as the algorithm proceeds, may be unsatisfy- 
ing to readers who would like more insight into the broader linear algebra 
concepts underlying the algorithm. Such readers should see the conver¬ 
gence proof given in Parlett (1980). 

Appendix C. Description and Use of Fortran Codes for 
Solving Problem LS 

A set of computer procedures implementing the principal algorithms pre¬ 
sented in the book has been developed. An additional subprogram, called 
BVLS for the Bounded Variables Least Squares problem, has been added 
since the 1974 publication. This set of software is described in Appendix 
C. This software is available via the Internet from the NETLIB site, which 
is also discussed in Appendix C. 

The NETLIB site contains a very valuable collection of mathematical 
software, including a number of packages relating to least squares com¬ 
putation. LINPACK [Dongarra, et al. (1979)] and LAPACK [Anderson, 
et al. (1995)] cover the basic problem types of linear algebra, including 
linear least squares problems. The block-oriented algorithms of LAPACK 
are primarily realized with Level-3 BLAS [Dongarra, et al. (1990)]. Also 
used are Level-1 and Level-2 BLAS [Lawson, et al. (1979); Dongarra, et 
al. (1988)]. The Level-1 BLAS package includes subroutines for both the 
standard (4-multiply) Givens transformation and the “fast” (2-multiply) 
Givens transformation as described in Chapter 10. The implementation of 
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the 2-multiply Givens transformation incorporates code to take care of the 
overflow/ underflow problems of that algorithm. 

As a follow-on to LAPACK, the software packages ScaLAPACK, PBLAS, 
and BLACS were developed with the goal of achieving a practical level 
of portability in the solution of fundamental linear algebra problems on 
distributed-memory multiprocessor systems. The first “official” release of 
these packages was in March 1995. See Choi, et al. (1995). All of the 
software mentioned in this and the preceding paragraph is available from 
NETLIB. 

The nonlinear least squares problem is represented in NETLIB by soft¬ 
ware described in Dennis, Gay, and Welsch (1981), Hanson and Krogh 
(1992), and More, et al. (1980). 

The Fortran 77 code QR27 [Matstoms (1992)] for sparse QR is in 
NETLIB but requires the code MA27 from the Harwell library, which is 
not in the public domain. 

Also in NETLIB are the package of MATLAB procedures for apply¬ 
ing regularization methods to least squares problems described in Hansen 
(1994) and the MATLAB sparse QR procedure SQR of Matstoms (1994). 

The integer linear equations and least squares package by Springer 
(1986) is in the TOMS (Association for Computing Machinery Transac¬ 
tions on Mathematical Software) portion of NETLIB. A software package, 
CUTE, for testing optimization software [Bongartz, et al. (1995)] will be 
in the TOMS portion of NETLIB. 

A comprehensive guide to optimization software, including quadratic 
programming and nonlinear least squares, is given in More and Wright 
(1993). 

The optimization packages LSSOL, for the linearly constrained linear 
least squares and linear and quadratic programming problems, and NPSOL, 
for the nonlinear programming problem, are described respectively in Gill, 
et al. (1986, 1984) and are available for a fee from Stanford University. 

The software package LANCELOT for large nonlinear optimization 
problems is described in Conn, et al. (1992). For availability of the soft¬ 
ware, inquire of phttimath. f uncp. ac. be. 

A library of C codes for linear algebra, including least squares, is de¬ 
scribed in Stewart (1994). 
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